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FOREWORD 



Weed For A Research and Evaluation Handbook 

There ajre events rapidly gaining momentum In education that dictate 
the need for the basic tools to perform evaluation^ of vocational programs. 
These events come generally under the broad heading' of Innovation or ^ 
educational reform of various types. These Innovations, In turn, are 
Intertwined with a milieu 6f other forces Impinging on the educational 
systems. These -forces Include such things as desegregation, collective 
bargaining, tax r^eform movements, bilingual education, a need to serve 
handicapped and disadvantaged students, declining enrollments, declining 
academic test scores, and a host of other Issues. Vocational education 
must operate Its various delivery systems within the context of these 
several vectors and Is thus heavily Influenced by them. 

Educational Innovations are often financed under various federal or 
state acts ^nd thus quite naturally carry with them a mandate for evalua* 
tlon. The presumed basis for this evaluation Is to measure the effects of^ 
programs on students and thus provide a basis for program Improvement, 
policy making, and decision making as to program continuance or elimination; 
In actual practice however. It appears this allegation may not always be so. 

Vocational education has in fact received Its Impetus from federal 
legislation for many years, albeit only a small segment of total expendi- 
tures nationwide actually come from this source. The most recent federal 
act (1976) however, as has Its two Immediate predecessors (1963 and 1968), 
requires evaluation In a broad sense coordinated from the state level^. 

This handbook addresses that type of evaluation but It also most 
specifically orients Itself to Individual program evaluations at the local 
level. It Is written from' the pierspectlve of a relatively large urban 
school district and primarily deals with high school-type programs, although 
nearly all of the principles and practices cited are equally applicable to 
post-secondary programs. 

P.V.F. 

« 

August 1979 
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INTRODUCTION 

This is a handbook that Is designed to' be of direct and practical use 
to local school district practitioners of vocational educatioji evaluation. 
Every attempt has been made to present the evaluation designs and instrumen- 
tation in a formthat is easily understandable and immediately useful. Most 
inolviduals charged with such responsibilities, outside of sophisticated 
university settings, simply do not have the detailed expertise necessary to 
grapple with .the- intricacies of complicated evaluation schemes. Indeed, the 
craft of educational evaluation is itself such a relatively new activity, 
even the professionals are still debating th6 outlines of the art. 

In any case, such detailed skill is rtot needed .when dealing with' the 
every day on-line delivery of programs basically designed, to prepare secon- 
dary school students to cope with the world of work that exists outside 
their school settings. That is not to say vocational education evaluation 
is any less important than other educational disciplines. It is in fact 
perhaps one of the most important,' and certainly the desired student out- 
comes are clearly delineated and thus capabffe of measuremant, in most cases. 
» 

Vocational education has long needed to define its own philosophy so 
that it may take its rightful place among other segments of the educational 
realm that are often more highly regarded by the public. It is suggested 
that a suitable philosophy for vocational education might be: 

An aoademia-eduaational disaipHne based on a body of aooicnulated 
knowleage ana fosteHng the development of new knowledge which is 
dtreated toward decision making, policy making, and skill acquisition 
as tt affects the development of human resources to deal with their 
employment needs within the economic context of their time. It is 
accountable for the product 6f its activities and is willim to exprea<i 
the results to the public with candor and objectivity. 

This manual is an effort to dear with the Iftter need for accounta- 
bility as related to individual programs. In additio i to being a basic ' 
handbook for evaluators, using a case study approach, it also includes 
a first section briefly dealing with the history, heritage, and legislation 
pertaining to vocational education and a third section covering a sampling 
of evaluation issues and practices, including its politics. Thus an 
additional useful dimension is provided. Despite the brevity of the over 
all handbook it could be used as- an adjunct text book for graduate level 
courses in vocational education administration and leadership. 
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PART I 



HISTORY, HERITAGE, AND PRESENT STATUS OF VOCATIONAL EDUCATION 

* Special attention Is dlrecte.) to the federally aided programs of 
vocational education; not only for the historical basis but also because 
this is where the actual commissions for evaJ.uation have taken root in 
recent years. The t^rm "vocational education" is relatively new in the 
literature of education; having come into prominence near the beginning 
of this century. Actually the literature in this field probably is on 
the short side in comparison with other fields of education. It is 
looped that it will soon take its rightful place in the annals of academe. 

The purpose of this background section is to provide the user of 
this handbook an ever so brief overview of the earliest beginnings of 
vocational education so as to set the stage for the ensuing pages which 
will place us into a position for having a base upon which to illustrate 
evaluation strategies. 

Vocations M individuals in our society have been classified In 
various ways and it 1* *ell known that a person* s vocation has an impor- 
tant bearing on his or'^ter^^stattis in society. Some vocations tend to 
--provide more pre stl^^ than pthers; are not necessarily related to the ' 
accompanying monetary rewards; and often change in popular favor through 
succeeding generations. 

Be that as it may, the concept of systematized employment preparation 
had to start someplace. This not being an historical treatise per se, 
it would be inappropriate to devote excessive space to the full gamut of 
development in the training for work. Suffice to say, it was not until 
the advent of apprenticeship, which in itself is very old, that we find 
distinctive patterns of teaching industrial processes. In a historical 
sense it was "only yesterday" that any systematic attention was given to 
vocational education as an integral part of formal educational programs. 

Apprenticeship began in Greek and Roman days and continued through- 
out recorded history. In the United States, apprenticeship indenture 
started in the 1600 's and continued until the 1800' s, at vhich time the 
Industrial Revolution created new demands and requirements for diCferent 
methods of providing trained labor. This ultimately led to a translation 
of the industrial needs through various typci of training schools. Manual 
training, and other similar educational endeavors first came into formal 
organized course structure in the last quarter of the nineteenth century. 



Historical Development Influenced by Federal Legislation . 

Shortly prior to the manual training concept a significant event 
occurred that has had profound and lasting effect on the cour.-.e of voca- 
tional education and particularly the federally funded influence. This 
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was the founding of Land-Grant sciools« \ 

The federal Influence is mentioned iTequcntly, not because it is the 
largest contributor to vocational education funding, because In fact it 
is not, but because its presence, through the legislative process has been 
very deeply felt. Vocational education itself has been the longest, con- 
tinuously federally aided educational program, of any in this country; 
dating formally from 1917. However, the one related idea of Land-Grant 
Schools has existed in a continuum even longer. 

The concept of national responsibility for education actually goes 
even farther back; to the Northwest Ordinance of 1787, but it was not until 
1862 that this .responsibility was formally initiated in the context of 
vocational-type training. At this time the Morril Land-Graift Act was 
passed. TYdn provided the impetus for each state to develop a college 
where the primary emphasis was to be in agriculture and the inechanic arts 
(although not to the exclusio^ of the liberal art^). For the first time, 
higher education was opened up to the tAasses of people, and for the firsc 
time there was to be the opportunity for individuals to major in topics 
C\, other than the traditionally acceptable courses in philosophy, theology, 
literature; and science. 

Also this concept was unique in that, for the first time, federal 
support was to be used far specific rather than for general purposes. 
Therefore, in the sepse that federal encouragement became a vital part 
of public education^ that "non traditional" subjects were to be emphl- 
slzed; and that the concept has been continued to this day we may safely 
state that indirectly federal aid to occupational training actually 
started in 1862 with the passage of the Land-Grant Act. 



Contemporary Development Influenced by Federal Legislation 

Starting with the Greco/Roman tradition, up through t-be various forms * 
of apprenticeship in Europe and in this country, the manual training-type 
schools, and on into the Land-Grant' tradition, we find ourselves on the 
threshold of a continuing legislative influence on vocational education in 
this country. 

Although federal support strictly for vocational education at pre- 
college level, goes back to the Smith-Hughes Act of 1917, the Vocational 
Education Act of 1963 is the beginning point for the consideration of all 
recent federal* efforts to expand and Improve vocational education. That 
Act sought to update and upgrade vocational courses by requiring the 
United States Office of Education to fund national research, experimental, 
and developmental programs. New funding was also provided to the states 
for the construction of area vocational schools and for work-study programs. 

The Vocational Education Amendments of 1968 further expanded the 
flexibility afforded the states to offer broad training progrnms by re- 
moving the narrow purpose by purpose state matching of federal expenditures 
required in previous Acts. Those amendments also sought to encourage the 
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States to evaluate better the appropriateness of the Job training they 
were offering and to coordinate be£ter their programs with other federal 
and state Job training prograas. In addition , those amendments sought 
further updating and upgrading of vocational programs by providing cate- 
gorical funding for exemplary projects, coiperatlve education, curriculum 
development, and special programs for the disadvantaged. Federal support 
'was also continued for the construction of area vocational schools and 
for work-study 'programs. 

I 

The states were enjoined to give greater attention to persons with 
special needs by requiring that at least 15% of each stag's basic grant 
be used for the disadvantaged and at least another 10% be used for the 
handicapped. The states were also encouraged to fund more programs for 
persons who had completfed or left high school. Lastly, these amendments 
sought to provide greater federal funding for poorer school districts and 
to Involve greater lay partlclpaf on Irf the planning and evaluation of 
the state/s programs by creation of the State Advisory Councils on Voca- 
tional Education. 

^ The 1975-76 Congressional hearings on vocational education showed 
that the Vocational Education Act of 1963 and the 1968 Amendments had 
clearly achieved tremendous progress. This can be seen In greatly Ihcrea 
enrollments at all levels, especially among postsecondary students and 
among the disadvantaged and handicapped, greatly Increased expenditures 
of funds for yocatlonal education from local, state and federal sources, 
the construction of thousands of area vocational schools throughout the 
country, an Increa^^e In the number of trained teachers, and an expansion 
In the number and variety of course offerings and occupational training 
areas. 

The hearings also focused on the need for Improvements In various 
areas. Many witnesses. Including members of State Advisory^ Councils, 
fl^tressed the need for better planning and evaluation; greater flexibility 
.on the part of the states; greater attention to the needs of the disad- 
vantaged, the handicapped, and women; and greater coordination of voca- 
tional education and other related programs. 

In the Vocational Education Amendments of 1976 (PL9A-A82, Title II), 
Congress recognized the Impact and momentum of .vocational education by 
extending the Act through 1982. It also made a number of revisions In 
the Act, designed to bring about Improvements In some of the areas Iden- 
tified In the hearings. The major revisions are: 

(1) Increased emphasis on planning and evaluation: The State must 
develop a five-year p^^an, an annual plan and accountability 
report, and conduct program evaluation. The State Board for 
Vocational Education must Involve the active participation of 
a number of Interested agencies, councils, and groups In the 
planning process. Including the State Advisory Council and the 
State Manpower Service Council. 

(2) Consolidation of categorical programs: With the exception of 
Consumer and homcmaklng and programs for persons with special 
needs, a.Tl program funding Is consolidated Into two blocks, 
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one for basic grants, and one for program Improvement. This 
allows for greater discretion on the part of the planners In 
designing programs to meet each state's particular needs. 

(3) Set-asldes: The minimum set asides for the disadvantaged 
(increased from 15 to 20 percent), handicapped il6 percent), 
and postsecondary and adu^t programs (15 percent), must be 
specifically matched with an equal amount of state funding. 
Federal funds for state administration must also be matched 
with state funds. A new set-aside for guidance and counseling 
is established, which is a minimum of 20 percent of the program 
improvement block grant. 

(4) Coordination with other programs: Emphasis is placed on 
coordination with related programs particularly the Comprehen- 
sive Employment and Training Act (CETA) . Local applications 
and the state plans must describe the degree to which voca- 
tional education, employment training, vocational rehabilitation, 
special education and other programs assisted under this and 
related ^cts are coordinated. 

Clearly, the federal influence has been large. Perhaps no evaluation 
would have occuAed tiad it not been for this general assistance. Even 
today, precious little real local program evaluation is undertaken, except 
for what is actually required through the various state *plans. The bona 
fide evaluation implications are one thing; the moral and geneVal program 
improvement implications, from the local point of view may be something 
else entirely, or may be incorporated within the stat# requiremenps. In 
any event there are implications for vocational education evaluation from 
several points of view; , not the least of which is the 1976 legislation. 



Evaluation Implicationa ^ ^ 

The permitted uses of federal funds at the local level plus other 
school financing problems presently facing many states could alter the 
course of program evaluations. However, the Implications of what must be 
done are reasonably clear. In line with the 1976 legislation the states 
must prepare five-year planSi, annual accountability reports, and conduct 
program evaluations. Generally it seems that programs are to be reviewed 
every five years; including follow-up and employer feedback. Different, 
states may interpret this requirement in different ways and may have 
varying degrees of what functions arc performed locally versus what func- 
tions are performed at the state level. 

Basically three types of evaluation are needed: 

Process . 
Follow-up 

Student Impacjt ^Individual Pro];,ram) 

The overall process Is best conducted from the state whereby an 
objective overview of statewide performance levels may be obtained. 
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Process evaluation of individual programs however , should be an important 
ongoing activity in, any local evaluation efforts* 

Follow-up evaluation may , be conducted by the state on a 8am()ling 
basis if it wishes. This approach is acceptable for obtaining a broad 
perspective of program ef fectiveness. It does little to help young peopli 
directly. It is better for local program improvement and for helping 
individual youngsters to conduct a well planned follow-up at the LEA ort 
district level, striving for a 75% or more response rate. . 



Student impact evaluations clearly must be conducted locally; The 
bulk of this handbook will be directed towards the techniques of local 
follow-up and local student impact. When possible, the use of process 
techniques will also'' be emphasized. 
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GOALS AND OBJECTIVES OF VOCATIONAL EDUCATION 



Simply Stated , vocational education has always beeii considered as 
education that prepares people to go to wor)c; especially in jobs that are 
relevant and useful to our society at some given point in time, or poten- 
tially useful at a near future point in time. This then, makes the 
measurement of student outcoiqf s a relatively easy matter. Are they pre- 
pared in the skills considered necessary by the occupatio " Do they have 
the appropriate attitudes for successful work experience? Do they obtain 
Jobs in classroom-related occupations? Do they continue in these jobs? 
Finally^ do 'their employers consider them to.be "goo4" employees that 
have benefited from their school training? 

The traditional view of vocational education is that it is a vital 
£prce in thte "wage competition" theory of labor market operation., That 
is, it assumes people come into the labor market with a pre-existing set* 
of skills (or lack of skills) and that they then compete against each 
other on the basis of wages. Education is therefore crucial because it 
creates the skills which bring. people into the market. ^ / 

Ah opposing theory is characterized by "job competition" rather than 
wage competition. Under this theory, the function of education is not to 
confer skill and hopefully higher wages for the worker; it is rather to 
certify his "trainability" and td confer on hira a certain stat4s by virtue 
0i thiQ certification. Jobs and higher income are then distributed on the 
basis of this certified status. To "the extent that job competition rather 
than wage competition prevails In the American economy, our lon^ standing 
beliefs about the eco^iomic and social benefits of vocational education may 
ultimately need to be altered. This is one of the reasons, alluded to in 
the^Introductlon, why v|ocational education needs to' haye its own body of 
research literature whith records the exploration in the hitherto generally 
accep'ted patterns of training. 

^or the purpose o^ this manual we shall assume that skill training 
does provide measurable benefits to the recipient in the job marketplace 
and that th^se outcomes are measurable and worthy of the effort to do so. 

'* . ' - * • 

A General Philosophy 

Vocational Education as such probably does not presently have a 
clearly defined philosophy. It does, have a set of basic* principles that 
remained virtually unchanged for 60 years or so. ^ A suggested generai 
philosophy yas put forth in the Introduction section. What is needed 
here Is ^a philosophy for evaluation. What are the reasons'for evaluation? 
Because the state or federal funding sources require it as a condilioh of 
granting and continuance process and product effects of programs? ' Per- 
haps even as a basis for decision making at various educational levels? 

A suitable base for a philosophy is that evaluation and inGtructional 
pvogrdm are one and the Game. Evaluation is as ruch a part of the prcjraip 



OS ia ,ciirrioulurrt'' developrnent and ola&'jvoom inetruation. In no other way . 
can the effects of programs be* documented and assistance be provided for 
liB|>rovement or curtailment Evaluation should be *'lov key*' and should 
Interfere as little as possible with the actual instruction process al- 
though It Is a vital part; of It. The personality of the evdluator'ls 
paramopnt. 'By gaining the confidence of program staff and participants 
and by removing the '^threat" of evaluation he can become a significant 
contributor to educational outcomes. Decision makers at both/state and 
local levels should pay attention to evaluation results; the truth is that 
they very often do not do so. As a general rule, evaluation as a profes- 
sion, has a ways to go before results are taken seriously. 



Does Vocational Education Help Close the Gap Towards Economic Equality ? 

This question arises again and again if one squarely faces the issues. 
Vocational educators are often inclined to accept as gospel, the concept 
that acquisition of a set of skills gives one student an edge in the em-i^ 
ployment market over another student who has not had such training. Some 
recent, well publicized research, has shown that the output of the schools 
(including vocational education) Is not related to what goes on in the 
school, or 1$ not related to equalizing opportunity to succeed ox fall, 
but is based largely on a single input e, namely the characteristics of the 
entering children. This theory proposes that schools have, absolutely no 
control over the two factors that weigh most heavily on success: heredity 
and home en\ onment. 

Assumin*" that vocational education purports to provide an edge, at 
least in the obtaining of initial emplo3anent, then in view of the theory 
that schools may. not be change agents for equal opportunity it is fair 
of us to ask: ' 

^^Ave graduates of high school vohational -programs who go straight to 
jobs (or related oormrunity college training and then to jobs) 
better off in the quality of jobs, earnings, unerjiploi/ment rates, - 
or job satisfaction than^c^frtparable nori vocational graduates who 
go to work -umediately? Is the nature of the curriculum a causal 
or decisive factor in any difference between these groups? 

This question raises a host of research problems that will provide material 
fdr countless doctoral dissertations in the future. To date the most 
coiiinon research finding has been that vocational graduates obtain their 
^first jobs more quickly and, subsequently, experience fewer and briefer 
bfislls of uneiAployment than others with a high school education. But the 
evidence is by no means conclusive. 

Again, for purposes of this handbook, we. shall assume vocational 
education can be a contributor to educational and societal equality * 
despite naggjfng doubts about what research has actually told us so f^^r. 
This leads us directly to the goals created Ijy the 1976 legislation. 
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Goals Established by 1976 Legislation for Evaluation 

The goals of vocational education - especially the evaluation thereof, 
as spelled out in PL 94*482, center around states* responsibility to: 

o prepare a five year plan 

o prepare an accountability reporj, based on the five year plan 
o conduct evaluation of all programs funded under the Act, within 

the period of the five year plan 
o measure student job success by tates of employment, wage rates, duration 

of employment, and employer satisfaction with performance of vocational 

education students as compared with those who have not had vp' ;tional 

education. 

o evaluate the stales^ entire delivery system each five year period by the 
use of valid sampling^ techniques. 

The goals appear clear cut. Evaluation of the federally assisted • 
program* by the statie is required ,and should be a district responsibility 
also . After all, how may these data be collected without the complete ' 
cooperation and assistance of loc^^l district? In subsequent sections we 
shall deal primarily with the local function and related techniques. The^ 
purpose of tMs section is to lay out in definitive terms the goals and 
legislative mandates to evaluate Vocationa^^^ programs. The moral and 
program improvement Imperative for ^he local deliverer of ijervices probably 
supersedes the legal requirements in terms of helping students and ulti- 
mately contributing to the literature' in vocational education. 



Interpretation oF^egislative Goals at the Local Level 

The goals as spelled out in federal legislation may be interpreted 
at the local level in. one of two basic ways, to wit: 1) What is required 
ami, 2) What is desirable for program improvement. 



What is Required 

/ " . • 

What is actually required locally, revolves around development of a 
plan indicating the 'nature of the federally aided program(s), plus an 

f ccountability report , and a follow-up of program comi:>leters. The new 
egislation also broadens the follow-up to include employers of program 
.completers and leavers. Traditionally in the past LEA's have been required 
to:' 

o prepare a plan. 

o prepar^ an enrollment report showing numbers of students in pro- 
grams, broken down by sex; dlsadvantagemeht , and handicapping 
condition. 

0 prepare a follow-up report approximately six months after students 
complete programs. and leave school (i.e. follow-up report in Decern* 
ber/January for. the previous June's completers- leavers) . 
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The new additions include the accountability report and follow-up 'of 
employers. Techniques of follow-up are treated in this handbook, as are 
other approaches under ,the general rubric of accountabili|:y. 



What is Desirable for Program Improvement 

What Is desirable, of course, Is for the local district to conduct an 
ongoing process and product evaluation of Its vocational programs. This 
could be done In concert with the federal mandate to evaluate each program 
(statewide) every five years. The states will no doubt attempt to assume 
major responsibility for conducting these evaluations. This does not ^ 
preclude Individual districts from doing their own evaluations, and most 
Importantly, using the resftlts of these evaluations for decision making 
purposes. ^ i ^ - 

The author was fortunate enough to be employed by a school district 
that assumed tl>ls responsibility. While decision making may not always 
have been based on evaluation results (because of inhibiting local condi- 
tions such as long-term entrenchment of programs, teacher tenure, etc.) 
every major program in the district was evaluated during a five year ^ 
period. The material in this handbook is based largely on ideas developedj 
in tfhis and other districts in the same state. A series of case studies 
are described to illustrate the use of ten basic evaluation techniques for 
local vocational programs. These techniques are illustrative of some 
standard evaluation strategies, merely applied in a voclttional setting. 



Study Questions - Part 1 

a) Prepare a twQ page paper describing your perception of 
the fdur major turning points in vocational education 
legislation. 

b) Contact your State Department of Education and ask for 
materials relating to their evaluation procedures. Write 
a two page synopsis.. 

c) Develop a chart that compares and contrasts statements 
regarding evaluation in Smith-Hughes, VEA 63, VEA 68, and 
VEA 1976. 

d) . Develop a timeline of federal legislative enactments. 

Analyze the various groupings that 6ccur. 
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PART II 
EVALUATION CASE STUDIES 



The evaluation illustrations are directed to three major foci, ten 
specific techniques, and seven actual case studies to show tlie use of the 
techniques. This is clearly a very limiting factor - there are far more 
than ten techniques that are generally used in educational evaluation. 
Our position, . is that these Jbasic functions - if mastered properly - are 
normally sufficient to provide the local evaluator with the basic tools 
he needs to provide reliable an^ useful evaluation information for voca-« 
tional education as presently structured in our schools. Some of this 
information may be utilized to complete the required state forms; but hope 
fully such data will frequently be used in process and program decision 
making. The basic techniques to be discussed are: 

1. Descriptive survey 

2. Interviews: student, staff, and employqr 

3. Self evaluation 
A. Follow-up 

5. Comparison groups 

6. Criterion testing 

7. Item^analysls 

8. Hypothesis testing and sample size 

9. Case study ^ 
10. Process evaluation 

These processes are described in a series of case studies which are 
llsted^by three areas of emphasis. 

%> 

o Area Wide - Case Study 1 

o School District Wide - Ca^e Studies 2 & 3 

o Local Program - Case Studies 4-7 
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CASE STUDY NO. 1: PROJECT FOCUS 



One obstacle to preventing efficient development and coordination of 
quality vocational education programs In various states, especially urban 
areas within states, has been a lack of "knowledge as to what constitutes 
an actual delivery system. At the time of the PROJECT FOCUS study the 
policy makers In at least one stdte appeared to be making their decisions 
in a partial vacuum. Communication between many of the deliverers was 
lacking; In fact no one really knew who. In fact, constituted the complete 
family of vocational education training agencies In any given urban area. 
Even the concept of defining what Is the urban area or metroplex presented 
a problem. 

Members of the state legislature, and many other citizens were 
concerned about the complexity of systems offering vocational education 
programs In the state. There was (and still la) some reason to believe 
there was a duplication of eflforts and services among the various educational, 
community service, and manpower agencies, both public and private. There 
was clearly a need for planning to ensure a maximum of cooperation and a 
minimum of overlap ot services provided. Th^ net result could be an 
effective state- level policy which would be based on underlying beliefs 
or assumptions. ' 

One of the principles of vocational education 'calls for a coordination 
among various agencies and Institutions to optimize the use of available 
resources. Since agencies, both public and private, often serve a different 
clientele, aave different goals, and may not have a desire to cooperate, 
the task of coordination becomes difficult. In any event, until such time 
as a reasonably valid picture of what exlstjfi Is described, it Is clearly 
difficult to attempt total coordination. 

In an effort to establish some baseline descriptive data thd State 
Advisory Council on Vocational Education, through a consulting arrangement 
with the Harrlsop Unlflod School hlsjtrlct, commissioned a descriptive 
study, of all deliverers of vocational training In the general Harrison 
Dlsirfct service area. 

Harrison Itself has a population of more than 800,000 people and is 
considered to have one of the best managed school systems In the nation. 
It Is overshadowed by at least one other larger metropolitan area In Its 
state although its growth has been steadily upward since the second world 
war.. It has a large accumulation of ''military facilities, but the industrial 
base has broadened far beyond this primary "industry" during the last 30 
years. 

The county In which the city of Harrison Is located. has about 1.2 
million people, but approximately 80% of those within Harrison and 

the liiunediately surrounding clusters of bedroom communities commonly found 
in the west. It was therefore decided to Include as the study area an 
urban complex that extended beyond the limits of Harrison and the Harrison 
school district to e^nbrace a population of about 1,000,000 persons, of whom 
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the majority of heads-of^famny, work within the boundaries of the study 
"tree. The boundary lines were drawn so as to coincide with the Inclusion 
of three complete community college districts. 

The basic techniques of the study were to: 

• Define the geographical area. 

• Identify the deliverers of training services; public and private. 
. Develop a questionnaire to supply the desired data. 

« . Conduct structured interviews with agencies using the questionnaire 
as a base. 

• Prepare a final descriptive report. 



Technique 1; The Descriptive Survey 

A descriptive survey is intended to describe a specific set of 
phenomena in and of themselves. The rationale for the purely descriptive 
survey is the fact that the information provided is in itself the answer 
to the research question posed. The most famlHar example cf the descriptl^ 
survey is the United States Census, which seeks information designed to 
answer a wide variety of research questions beginning with "how many 
people...". Many descriptive surveys have this census type head-counting 
aspect beginning with "how many," "what," or "where." For example, a 
descriptive survey would be appropriate to seek answers to questions like 
the following. "How many classroom seats are there for retarded children 
in the Northeastern states?" "How many teachers currently teaching grades 
4 through 8 have less than a baccalaureate education?" - "What is the median 
salary for full professors In fully accredited colleges?" "Where are the 
colleges and universities that offer advanced preparation in educational 
research?" 

Thus, in educational research or evaluation, there are two conditions 
which occurring together suggest and Justify the descriptive survey; first, 
that there is an absence of Information about a problem of educational 
significance, and second, that the situations which could generate that 
information do exist and are accessible to the researcher. 

In the case of the FOCUS study the descriptive survey ended with the 
description Itself. In another case Study we shall demonstrate how the 
descriptive study, when appli^^.d to a single school or vocational program 
can have a definite Impact on the process evaluation of a project to help 
keep it on the course that was originally Intended. 



Advantagea of a Deecriptive Survey: 

0 

. a. ' Information provided is in itself the answer to the question 
posed. ' 
b. Documents a particular situation, by person (s) actually 
witnessing it, and thus becomes valuable information for 
possible later historical research. 

* ■ * 
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c. Provides data far process evaluation to Improve programs as . 
they currently operate. 
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Dieadvantagee of a D^ecriptive Suxvey: 

The material presented may be very subjective and subject to 
bias. 

bf Analysis of data is usually not planned until after it is 
collected (This may not always be a disadvantage). 

c. Sample described is o^ten selected on the basis of convenience 
rather than randomly. 



Technique 2; The Interview J 

The interview can be done in a number of ways, but perhaps it^ best 
use is in a structured manner; that is in conjunction with a previously 
prepared questionnaire. In the case of FOCUS, the questionnaire was pre- 
pared, then the potential respondent was contacted for an interview 
appointment. About a week before the appointment a copy of the question- 
naire was mailed to him with a cover letter reminding him of the upcoming 
meeting to liscuss it. The ideal situation (infrequently attained) was that 
the agency would have the questionnaire completed prior to the arrival of 
the Interviewers. Two interviewers wert used so as to reduce the possi- 
bility of subjectivity in interpreting answers. Ideally the interview it- 
self would consist merely of clarification of points that may not have 
been completely understood in the questionnaire. More often. than not the 
inter«fiewee was not prepared ahead of time and the interview consisted of 
a detailed discussion of each applicable item, thus unproductively utilizing 
the time of all parties, A copy of the questionnaire which was used follows. 



Framing queetions for th^ interview: 

a. Ascertain whether the population from which the respondents have 
been selected actually has the information sought by the Interview 
and that the questions permit the reasonable recovery of this 
information. 

b. Questions must be framed in language that ensures effective 
communication between the interviewer and the respondent. Omit 
all ambiguous vocabulary. 

c. Make certain respondent appreciates the purpose of each question 
he is asked. Av6id arousing any suspicion or resistance. 

d. Avoid leading questions (questions which suggest a desirable or 
preferred answer). 

e; Ensure that the frame of reference surrounding each question is 
clear so that each respondent hears the question in the same way, 
avoiding misinterpretations. (The two interviewers can help here) 

lA 
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f. Pretest .the interview in a pilot study to eliminate weakness and 
experiment with alternative items o^ techniques. 



Advantages (aver a queetioi^aire) 

a. Permits greater depth. 

b. Permits probing to obtain more reliable data. 

c. Makes it possible to establish and maintain rapport w;lth respondent 
or at least determine when rapport has not been established. 

d. Provides a means of checking and assuring the effectiveness of 
communication between the respondent and the Interviewer. 

e. Ensures a larger response rate. 



UHeadvantages of an interview: 

a. Interviews are costly, time-consuming, and inconvenient. 

b. If the researcher takes advantage of the interview's adaptability, 
he introduces the problem of subjectivity and personal bias. 

c. Eagerness of the irespondent to please the interviewer, a v^ue 
antagonism that sometimes arises between the interviewer and the 
respondent, and the tendency of the interviewer to seek out answers 
that support his preconceived notions all complicate this method. 

d. The thorough training required of the interviewer to offset the* 
disadvantages becomes an additional burden, increasing the cost 
and time factors. 
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NAME OP AGENCY 



DATA GATHERING FORM 



* A QUIDS FOR THE STUDY CP 
HARRISON "MgrROPLSX" AGE»CI£S RSSPONSIBI£ 
FOR JSLIVSRIKG VOCATIONAL EDUCATION 



Adninlstered under the auspieea of 

The Advisory Council on Vocatio nal Education 
and Technical Training 

1975-?6 
(CACVE nLCT FOCUS PROJECT) 



\ 

\ 



This Instruaient should be completed ohly \iy a 
person or persons familiar with the overall functions 

of the agency* 



♦♦»♦»♦»♦■■ WW MM ■ ■ ■ ■ ■ | | »»»l ll »I IW»lWII »»«l . «lia MK II » liW»» 



EXPLANATION 



Kany people in this state are becoming increasingly concerned with the 
complexity of systems delivering vocational education services* Thtfre ' 
is some reason to believe a duplication of effort exists among and between 
the various educational agencies* To ascertain if this situation really 
exists, the Advisory Council on Vocational Education and Technical Train- 
ing (CACVE) has made the decision to take some definitive action. 

In order to develop a framework for analysis of the vocational education 
delivery system, CACVE has commissioned a study of an urban metroplex and 
the agencies that exlst*wlthln it* In addition to the usual public educa- 
tion systems, the study includes private schools and agencies under the 
purview of state and federal offices* 

Harrison was chosen for this effort and the area of study includes i 

The Harrison city limits plus the immediate adjacent 
high school and community college districts* 



^ THANK YOU 

(See last page for name and address of primary investigator* ) 
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DKI'INITIOfiS 



Tlie rullowing wordt; iiro ased in the questionnaire and are dofircd its follows; 

1. Administration : An educational function winch ini!Uid<\s |x»r^tf)ns iintl/or dollars 
d(?vo((Hi principally to tlu» dir(x:tion and ln;ma^;cn)lM^t of :in ai^.'ncy, including 
alsf» tiK! ncc<?s.«-^ii-y sup|)i)rt scrvicos; as op|y)stxl b> Lh<i simy i.vjk? of rcsinirccs 
allcKxitcti to insLiiicl.ion, p:uid:mce, olc. > ^ 

2. Adul t : A student, 18 years of ago or over, who is not in aWej^ilar hir,h 
sc1kx)1; excludinij a^;oncLes such as KETir. 

3. Apprenticeship : An approved training program for a reaignized trade. 

A. P^try-level : Skills needed for a beginning job in the occupation being 
described. 

5. Ouidance: An educational function which includes persons and/or dollars 
allocated principally to assisting students in milking informed ctoices 
relative to caj'CH»r and job selection, preparation, placanent, and jidvancement . 

1) Cbiinseling : A sub- function of guidance; usually defined as 
a close one-to-one relationship wilh students. 

2) Pluccmont : A sub-function of guid:mce; relating to counsel- 
ing iind/or other activities to help students bcctxne employed 
in situations which fit their needs. 

11. 1 hind i capped : Persons with physical, mental, or emDtional impainnonts. 

7. H ow many i)repai'ai for labor market : Nunber of students who exit the program 
with job-reiati-d skills. 

K. Out-of-Sc|vx)l Youth: High School age youth who are not enrolled in a formal 
.scIkoI prognun. 

9. I \)sl S e.x:ondary: Ixlucutional services of]'ercd beyond soconckiry cckicalion or 

10. IlL^lrain : Ti-aininn lor people u1k) obtained their orij^inal job skills in the 
past ;md who Irive been out of the lalx>r nm*ket. 

9 

\\. Sjxx!i ali nation : lYainlnt; above the^ntry-level in proj-^-nms ctesipnated *\s 
Uvtuncai or sctni-prorossional . 

12. Studen ts Sorvod; Total nimilx^ of students enrolled in a pro^i;ain during; the 
past one-year iHTiod; includes llK)se who have reccMvcil instnuaion ajui/or 
KUidance , and. phu'(in<nit services . ' ^ 

13. Subunit : /\ny local ion wlicro occuF^nl ional training luul/or r(?latod functions 
t.iJ;c place?. O.K. ju-lvK)!, outn»acU cc^iuer, store fronl oporalion, cnaiUinity 

'center — (rK\Ptc:il, finimcial, oi- icaai I tr;ido outlet , industrial site, i-lc. 
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\ • . . . ■ ■ 

i - ' 

. iA. UndupUcatod O^u nt; Niimbor of indi vidual students servcxl; regardless of 
I low nuiny sc|):uato proKnmis thcy'nuiy luvo been involved with.. 

I ' 

15. Upgrade ; Rirthor traininj; .to upRrado current skills to moot dhanRinj; job 

I or nuirkot dcminds. 

/ 

/ ■ • ^ . . • , 

i 

DE SonrriON of dhi.ivi-ry AGi-ycY - qui-stiqns 1 -7 

Apprcpcimate number of subunits in your agency that <?re located within the 
study area (e.g. three cxitmunity college campuses; number of CETA training 
sites, etc.). 



2. P3ea.se list, by naxnt;, all your subunits and then. check, in the appropriate 
boxes, the naturo of th'Q services pi'ovidcKl in each. 




FUNCTION OR T^TE OF 
SFRVICE PERTOriMED IN SLT3UNIT 



ERIC 
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3. What geographical area does your agency^ serve? (e.g. County of Sa^ \ . 

Juan; a specific school tilstrict boundary; a city limit, etc.) For 
your convenience we have Included a nap of Southwestern County on the 
following page. Piease Indicate your regular service arei using a 
colored marker. If you routinely draw students from outside this area, 
■ . please Indicate those boundaries with a different color and identify 

the difference in the space below. _ 

O ■ 



To what agency office do you report your enrollment, financial, and othei 
statist Icpl data? (e.g. State Department of Education Regional Office.) 
(Please attaok a copy of blank report form(s).) 



\ 



4. 



5. What do you ascertain to be the major educational funct Ion (s) and/or 
''missions" af your agency? (Check all applicable Items.) ^ 

Job entry level training • Other (describe below) 

Upgrading job skills 

Jbb retraining ' 

Apprenticeship training - 

Adult vocational 

training ; 

Job specialization . 

training > ^^^^ 

Fehabllltatlonal services ; ^ 

Services to out-of*. >v 

school youth 

Ihplaat training - 

Military job training ■ 

Vocational and career 

guidance : ^ " 

Counseling - — 

Job development [ 

Job placement - . , 

6. Do you contract with other agencies for facilities or services? Yes No 

If Yes, please describe the arrangements; Including numbers of students 
using leased facilities \ 

7. Do you lease facilities t-o others, or supply services? Yes No 

If Yes, please describe these arrangements, including number of students 
involved ' 
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DI-^ ^CRIin'lON OF STLDIXI S SiVm D - QtJI-STlON.S A A.NT) 9 

K. Hovv Rtudonts are served by your ajjency. Cluvk aJM apfUi cable iums in tho 
first coUimii; cnli-M' niunhcrs ol' students .st.'i-vui m coliiniiis 2 lhroup;h 9; anil 
. dv.'Ck lyi)0 or si.Tvio' in columns 10 ihrounh 20. (PloasD nolo definition 
of "Students St-rvixi") . 



WE OF CLinrr 

X 






tndtT 

f 1 u 

IH 












Instruction / 


' h Mr/A 


2 






s 




NM 


M 








• 


a 
0 




Ao/U/Vj/l.Vlt/ 15 /.lO / 17 /18 1 






























1 










Itmt Socondary 




























TT 

1 










ts:^ 

Adult Educ. 




























1 
























4 










*• 




, 1 












^Aipnutlri; 




















i- 






1 

* - 








: 1^— 








> 










1 — ' 






[— 


1 
t 


I 
1 




1 i 


Hi'rvibiliiut ti)n 
















i 


L_J 


1 


1 


























h 




T 

1 








































1 — ^ 


Youth 






















1 ■ 
1 

r 


•1 

■ i 1 




































! 


(InplAiit) 
















■ \ 




1 


1 

_i 


. . : , 1 i 


CXhor (Plon-rf*. 












> 






\ 

1 

j 










1 


J 
i 

1 . 
• 




ltnAI5 






[• 








1 




1 




I 


r 







^IUttl« tn bliA'kii 1 und 2 srtnild U- 



9. PlcjLso fiive us ytnir Ik^sI t^sLinuite ol* an luidn pl urat (;.d count to include all 
students rcix^rtttl in Qiu\sti()n H. ..(Please <*xplain il it does not equal 
nioek I Total in Qu(\st ion 8) * ■ 
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DESCRIPTION OF FUt^ING AQ^'CY CarTACT nraj\TIONSHIPg - QL'ESTIQNS 10-15 

10.'^ In the table below, plciuse estimate as accurately as possible, itsiiig your 
*o\m accounting systan, the sources of your financial siHjport and cat(?^^o^i2:e 
. by "puriXDses" for which used; (if you are unable to givf? a reacunably 
accurate breal^dowi in dollars - as between ,puri3bses - then please designate 
as a " percent of total expenditures.") ' . 

♦ ' -J 
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^ ^ 




111(11 USll) 






I 

I rut ion 


liisliue- 
t ion 


oui^Kjrt ' 
S**rvi(n.*s 


I 

AlhAv- 
aiu!cs 1 ♦ 


1 

(juiiLuiee 


v. 1 

!*laccii».*nl 
1 


V 

Cliipilal 
an lav 


OUieV 


Vli\-A 






* « 












VL\-U 


















\1j\-C 


















VI;j\-U 


















WA-V ■- 




% 1 














VL\-G 












9 


1 




VEA-II 


















CLTA 

1 1 i i 1.' 1 


















Title. 1 •">' ■ 










> 








Title 11 












r 






Other 




• 










• 






















Other Federul 
















:yr^ 


State 
ADA (si^ocify) 

• 












• 












/ • 


« 


















r 










Grunis 
ft . 


















Other Private 


t 








V. 


1 







♦I * StiiX'nJs paid lo trainees. 
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1.1. Which of tl-.c following? itims will your present accounting systctn allowyou 
to calculate? (Please chf?ck. ) I< 

Cost per student trained 

Cost |x r AI")A . . ^ 

Cost ix?r instructor hoiur 

Cost per phicanent 

Cost for plucfiiH^tit .sorvicoR 

Cost io V Kii i lUuK r St »i'V i cc»s 

• Q)st of udninisLnitive services 

^ Cost of ajiciJlary services . • 

" Cost for student recruitment " - 

Other (sixiciiy) ' 



12, In "what manner is follow-up data on former students recorded? (Please check.) 

Ninbor of cofniileticns ' . 

Numbt^r of cocnplot(xl and working 

Number lefl eai'^y for emiJloiitiont 
• Number working in area of .training ' 

Number vvorking in related job 

NOndxir continuing c<h.ication iuid training 

Placement data by pn)gram . ' 

Other (Kixjcify) , 



« 

13, In what'v,ay, if any, is yoiu* agency administratively and/or fiscally 
responsible to any other state or federal agency?^ (Clieck agency and 



describe relationship, ) 

Post Secondary FducaLion CcxnniB.sion 
"bepartmcnL ol lllucation 
VlvuKTllor's OfHcv (axnminity Col legos ) • 



Criminal Juslico Un}arlnK;nL 
"r}Dplo>iiKjnU iX^veloiJcir^nL Department 
J>ipartnx?nt ol I4i)^'>r 

jOther (plcofic name) 

"None 



Briefly descrilK? b^low wh-it this relationship is and what ''lunhrella*' or sUper- 
vifiory functions this apency pc^rfonns rolalod to yoiii* o\mi unit(s)* 
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14. What ajjoncies or entities do your profctision»'il st:il f mintain regular 
camnmication u'ilh? (OlrocL ccrmiinicution on a ro^iihir basis). Those 
mlKht tncludo "Hoalth M:in|»\vcir Comnittco" or - "The Jndusti'y /Education 
Council," etc. Please list. < 



NMQ-: OF AQNCY OR ENTITY 


State 


Federal* 


Private 


i — 






« 








t 




r 




• 



























1^ ' Please" place itans from Question 14 above in rank order of qontacts according^ 
to the "influence" exerted on your oix?i*ation. Also provide the tyi)e of 
information e.xclianged. ■ ' ' 



NAME 


' ' ■ ^ 

WIIY lUNKED IN 


TOE OF 
INFOIIMATICN EXQIAN'GED 






































r 
• 
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DEseniiyrioN of mMinxfi pimiv.is oninuD - qlistions ic-i 7 . . - 

16. In the table below, please chock in coluniis 2. 3, 4, 5, G, nnd 7, if you offer 
proiji'anis uiidor lht> broad job clusters as slmii. Please clieck tlio "n;mio of the 
program (not individiuil courses). Your particular prc^,Tniii nuy not luvo tli(? . 
exact title as sixjuii. Ti-y to fit it as closely as possible or,. if necessary, 
write in one of the blank spaces. Please enter in coluiiuis 8, 9, and 10 the 
number of students you exix^ct to be served tins year, aiid in column 11 .the 

5 number of students prei)ared for the labor market. 



t 

i 

ClLSTER/PROCniAM' 

' 1 


LEVEL OF Instruction , 


NUMBER OF 

STUDtMs sm\w 


gs 

11 


/ //^/ J" A'^ / 

/ 2 / 3 / 4/5 / 6/7 


• ' m 

^V/ //'/ 


I- AGRiaLUl{E & AGUnuisiNliSS 
Ap'icultural Dusiness 
Afp:icu It viral Rt?i3ouroes 
Agricultural Mechanics 
Agricultural alienee 
Animal Health jrcchiioloL»y. 
Forest Tcclinolo^iy 
Itort'iculturo Tuoluiolo^y 
Ornamental Horticulture 
Production Agricultuie 








1 


• 




• 




1 

1 
1 

• 




2, UUSINISS /\XU OtTIOi 
Account in 
Bookkeeping;. 
Dusinoss Michlnes 
Business Mina^anent 
Clerk Typist 
• Ciinpviter Prcx^irmnnin^: 
Cbvirt t Cbnf. Hv^ioriinu 
Data ht)cess in^; 
Lt^al Ass is time 0 
Secretarial Science 

* ■ ■ i 

• 




« 




t 












1 
1 



Ij Fij^ircs placi-d in coliuiuus 8 and U \wy or nay not bo (niual due to c.\pluratory 
Pi-ojn'aiTis, drojx^uti;, hiro-outs. rcquind classos, etc." 
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IC. (Cont.) 



tXVJL OF lNSn;l!CTION 



4. 



ausmin/T'ioQivM 



1 



KirmKii OK 



Is 



Atuliuvibiuil Vc\:luK>hn;y 
Qjiinercial Art 
Graphic Design 
Journal inn 
Letterpress Printinf; 
LitlKH^i-aphy (PIdio Offset) 
PhQtOi^iapliy 

Printing and Publishing— 
^ Admin iStrat ion 
Printing and Pul/lishihg^- 

Opc rat ions. 
Product DL*6ign 
Publications TechnoloRy 
Ridio Uroudcastin^i 
Tociuiical Illustration 
Tech. Materials Publication 
Telecasting ' ^ 



CaiSTiiUCTION \ 
Architect ui-al Draftit^ 
Bricklaying and Misonry 
Building Construction TCch, 
Building Trades 
Carpentry 
Civil Technology 
Electrical Ttvlmoflogy 
Heavy l.'iiuipit'nt 0|H'i';itions 
Pliribing and.Pi|x. lilting 
Sheet Metal 
Ti;ule'Llcx:tricity • 



5. UiVIlCiM^II^T L NAll.UVL tUSaJKCVS 
Electric l\)^ver anci*UisiriUilinn 
Eviroiunental Health Tinh. 
Tori'st PiiKlucls Tix hnt>h)i:> 

Mu'liK' Xt> luK)U4:y 
Kalural tU'SDurces Vanagamiit 
^ Soil Gmsrivation | 
Water c^uil ity (Jc^ntrol lixiU 
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Ui. (ami.) 



7. 



C3iJi;l•l:u/J'l^xln^^: 



1 



NlROUiH OK 



ti. iiij\j:ni 

Ui iilu 1 i st'.mcc ? 
iientul lly\:i(he ^ 
U»nl;il Lib Ti'< »uv)loi:y 
Ihniltli Care V.m;iK<riH'til 
InlLilaiiiin Therapy 
)lktJical Assisuini*!' 
Mttiiciil RO(tor(l TiH'linol(;i;y 
fcktliiuil Lil oratory Ttvh. 
Iknilal Tim;IuiuU>^ * 

Nursing Assisi;uu!i* 
Opi»ralin^ lixim Tiuljiuihi^y 
Optical TiH'linolofO' 
Physii'al TiKTapy Asstslujictfr 
1 'radical Niu'bin^; 
Itultuluuic 'rcrt'luinlo)^' 
IUi:ist<;rLtl Nui'sin^; 




iia.u: iximiics 

Cliild Cai*(* Assist ancL* 
Ci«iiii.*rcial Ikiklii^ 

UjI inary Ai*ts 

C\js|tin Apparel DesifOi & 

Tailoring: 
llii*U*iic T<'cl\nolot;y 
liryi.linitiinK 
F«jiid IU.'rvjcri^s — Qx>Uin^ 
\\%%\ ScTvu-o MinajMii'tit 
\\^\\ StTVici; ht^paralum 
InltTior U*simi 



IVVMilAClUilNc; 
Air OitidU ioiiini; Id^fri^- 
• Tal ion Ti>iuu)loj:y • 

i1iiinti*:ii TtrhnolcHCv | 
EliH'tnrtf.Hiiuuuia I Tt vluiohHcy 
l::li\:trutncs TH»luw)l«>Ky 
l»\innlurc UpliMsirfy 
Imhisirial StiiKTviriiDiis 

Omlrol T« i:linolni;>- 
Itit'linu* )I CV^ r \\\\\ 
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IG. (Oont.) 



dUSraVPHOGRAM 

1 * 


LLVii, oi* iNsrfRUcrioN 


m.i\m OF 


ii 

11 


M /MS /i 


/ c 


M 

/ 7 


' M 

/ / 9/^ 10 


8. (Gum.) 

Iticlnne Tool Tc\:Iuk)1(h^ 
fcloclunlcal Uusi^^ii Tui:Iuk)Io( 
Mcctunical Dritfiin;; 
Mcclonicul rn*aducLion Tcclu 
Ikjtallurtical Tccluiolotjy 
Numcrical-Control Machine 

Teclinoloyy 
Quality Oontrol 
Welding 


cy 

\ 


1 

i 








> 








• 


9. JIARKEFINa XiD DISTRIbl/naN 
Fashion Merchandisini^ 
Finance 

Fcxxl Distribution 

Hotel atxl Nblcl Assistance 

Hotel /Motel /Kestaurant 

9iLuiaK(n)cnt 
Industrial Mai-kcting 
Insurance 
Marketing 

Mr^i*r* K.1 ivl i Q i n fT 

Pui'diasin^; 
Itcal Estate 
Traffic JUnagawnt 


U 

• 




r 
« 


1 








• 






10. MLaL\NIG\L niADVS u SIJCVIC 
Air-Cbndiiioniiiji & IU,ar it- 
eration Servicing 
Fluid I\)Ukn"MiiuttMuuK*o 
\kvv A|n>li*UHV Sorvii iait 

iiiull>Ljigine ilLiktir 
* Stationary DiginLt;ring 
Watch Ri-^pair ' 


I' 

f 




• 











0 






11 • PUiSONAL siJivias 
Uoi'bor Training 
Oeauty Culture 

Mi>rtuary Science k Funeral 
Service 

• 

• 








1 
\ 

\ 

1 

1 

1 


r 


1 

1 








• 

1 

• 
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IJA'li 01* IN'.S'niU.'nON 


mr.\.\{ OK 


B 

■A' 

11 


/ 2 / 3 


i I J , . . - . 

\v-T /i- yy^ / J 

/ 1 /, 5 / f» / 7 


A / / 


12. miUC SiTviiH S 
(iinxvlions 

Crinur Libunitory lW'hin)l(>u> 
Vivo PrutcrHon TcH luu>l<>t:y 
li&w >^U'urcifnonl (I>>lic(> Sc. 
Library T(x:hnica1 Assisijirtt 
' Six* la I Work AssiKt;m(-i» 
Teacher /Vssisiance — ii|>i«cia1 

lulucat ion 
UrUin Plajining Assistance 


) 

.e 

I. 


\ 




n 


• 












Qinnifritial Music 
lfa.vTi*;iliiin l£?ndcrslup 
llM.utre Arts 








- 














AArix>s|.)«K:c TeduiolOKcy 
Aircraft Asscinbly * 
Airplane NVxiluinics 
Air IVulfic O^ntjh)! 
Autcvriibi U* Flixiy t^rpnir 
AuliYTDbile JUvlVAnics ^ 
Aut(m)tivo Tocl\nc)lbfr:y 
Aviation ^macinx^i 
Diesel Niectuiiics 
^Pllot Trainini; 
lVansix)rtation EnKimvrinji 

Ttvlttit>U)i;>' 
•fiHVlalri vv«i; 










• 






• 




« 


ir>. APPRhXrKISllIP 
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17. Plea«e describe the following Items In your own words. 

a. Where you feel there tiay be duplication or overlapping of 
, ' activities and/or services In the geographic area of thlff 

study. " . 

b. Are there any particular constraints or inhibitors that 
restrict vocational services to students, e.g. .financial, 
laws, reporting, conmunlcating, etc. 

c. How do you feel reporting requirements (e.g. as VE-48 et al) 
can be improved to better describe your programs and services, 
or for evaluation purposes? 

d. What criteria have you identified to measure program success? 
(If available, please attach copy of a typical evaluation 
report and/or your criteria.) 

e. Please describe what methods or meohanlsms are utilized to 
accomplish student placement into enploymjent. 



THANK YOU FOR YOUR HELP 



Dr. P.V. Foster 
Harrison City Schools 
4100 Exeter Street 
Harrison CA 
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CASE STUDY NO. 2: SITE VISITS 

Since states are required to collect evaluation Information frpm local 
districts they have chosen to accomplish this task In a number of different 
ways; few of which as yet deal directly with Individual program student 
outcome Information, ^e one partial exception to thls^ Is the annual 
folloyrup report of progfam completers and leavers. That method is discussed 
in Case Study number 3. 

During past years In a western state, under the stipulations of the 
Vocational Education Acts of 1968 and 1976 action was In a series of site 
visitation to 20 school- districts or regional occupational programs per 
year. The districts surveyed, selected on a random basis, varied from 
small unit's with one high school up to and Including the second largest 
district in the state with 15 high schools. The self -evaluation technique 
it employed in this activity, and the entire procedure itself must be 
considered as a process-type evaluation system. 

I. 

The final product ends up as a descriptive suinnary of district 
vocational programs based on a functional app-roach. On forms supplied 
by the state» the district described within certain guidelines, the exist- 
ing condition contained in the framework of the 12 basic functions designated 
as parts of a* total vocational education system. While states* modus 
operandi have changed somewhat under the 1976 legislation, the basic princi- 
ples of vocational education have remained relatively constant, as have the 
12 functions, which bear reiteration here. These functions are dlagraaned on 
the following page. 

A sample of the funn that has been used for this descriptive process 
is included here, however it has been subsequently outdated by the 1976 
legislation, and is under revision; but the basic concepts. Are still valid. 
The end' result is still moot in many states, although the district program 
must be an Integral part of the process of* self evaluation and program 
review (note in the example, that review and evaluation are central to the 
total process and^ impinge on all other activities). 

Technique 3: Self Evaluation 

An Important part of the self evaluation procedure Js a validation 
of the analysis by a "verification team" of practitioners from outside the 
district being reviewed (the term "evaluatioi/* is not used in this proce- 
dure). The district is provided with, sets of self-study forms several 
weeks in fidvance of the team visit so that they may review themselves and 
place a "grade'* upon their precelved performance in several areas of 
activity. The purpose of the validation team is to spend three days in 
the district interviewing teachers, students, administrators, school board 
members, advisory committee members, businessmen, and other lay people. 
The districts' own self evaluation forms are carefully reviewed and analyzed. 
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VOCATIONAL EDUCATION FUNCTIONS ANP ACCOUNTABILITY 



A. Needs and 

Opportunities 
Information . 



Population 
Needs 

Job Market 

Job Performance 
Requirements 



o 



0 




B. Planning 



Program Planning 

Vocational Educa- 
tion Promotion 

Student Recruit- 
ment 

Curriculum 
Resources and 
Ancillary ^ 
Services 



Student 
Services 



Guidance and 
Counseling 

Placement 



D. Instruction 



Vocational 
Instruction ^ 




EDUCATIONAL ACCOUNTABILITY 



Program Review Evaluation 
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SELF EVALUATION 

PERFORMANCE REVIEW - EDUCATIONAL ACCOUNTABILITY FUNCTIONS 

Instructions to Respondent: Please carefully consider questldns In Part E-*l 

before rating performance on each of the follow- 
ing criteria and completing the narrative Nummary 

on the reverse side. 

•t 

PERFOPMANCE 

CRITERU 5 r 3 2 10 

1«^R0GRAM REVIEW: Reviewing current and proposed vocational education offerings 
in relation to the total school program. 



s ' — 

The district: 

1 

a. Establishes procedures for the review of 
.all existing vocational education offer- 
ings and for consideration of proposed 
new offerings. 










r 




b. Develops evaluation criteria for decid- 
ing to continue, moc^^lfy, or delete exist- 
ing vocational^ education offerings and 
for determining whether new offerings 
should be added. 








* 




\ 


c. Relates Its decisions to areawlde progran 
planning, in the Interest of coordination 
of resources and services. 















2. EVALUATION: Ongoing and periodic assessment of how well the district Is 
performing Its vocational education functions, as« a basis for program Im- 
provement an^ management decisions, both locally and at state and national 
' levelsv ^ 



The district: 

a. Establishes and fixes responsibility for 
evaluation of the processes and products 
of the vocational education system on a 
continuing and periodic basis. 














b. Provides for the periodic evaluation of 
the district's performance of the 12 
functions of vocational education. # 














c. Obtains and Interprets evaluative data oi 
a continuous basis for use in adjusting 
system operations. 


i 
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PERFORMANCE 



CRITERIA 

• • 


5 


4 


3 


2 


1 


0 


d« Provides resources and services to assist 
district vocational education personnel 
In the development of measurable objectives^ 
evaluative criteria » and the oA*-|^olng 
collection of supporting evidence. 


• 












e. Provides resources and services to conduct 
periodic evaluation studies. 












c 


f. Provides fpr the collation and dissemination 
of evaluative results. 














PERFORMANCE SgO^E: 5-We are proud of this. > 







4»Mo8t of our objectives dre being met. 
3«Thls is less than satisfactory.' 
2«ThiE area needs attention'. 
l»We are not providing for this.^ 
0"Not applicable. 



0 



\ 
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At the conclusion of the visitation the validation results are orally 
presented by the committee, including a series of commendations and 
. recomnendations relating to the 12 functional areas. ,A true self-evalua- 
tion then, Irf conducted along structured lines relating to certified > 
objecti ves, goals, or outcomes and is validated by a group of outside 
experts in the fiel d being reViewed . There are sdme common sources of 
error that may introduce themselves into a self-study-validation process. 
They are: • 



The Halo Effect . 
). 

This is the tendency for an irrelevant feature of^'a unit of study ' 
to influence the relevant feature in a favorable or unfavorable direction. 
Typically, a strong, initial positive or negative impression. of a person,* 
group, or event tends to influence ratings on all subsequent observation. 
Impressions formed early In a series of observations often affect later 
observations; or impressions based on high or low status attributes of 
the unit of study affect observations on* unrelate?Ua±tributes^-quoting a 
celebrity's opinion on an" educational issue or assoSating emotionally- . ' 
loadedaabels with candidates in a hotly contested school board election. 
The more vague and impressionistic the variable to be rated, the more 
powerful Is the effect; the more specific and clearly defined the variable, 
the less evident is the effect. , 

' ( . ' . 

Rating Errors 

In addition to' the halo effect three other tendencies plague the^ 
validity of .ratingsi 

1. The over-rater error— rating subject in general on the side - 
of leniency or f^vorableness. 

2. . The under-rater error—rating subjects in general on the side 

of severity or unfavorablen^ss. 

3. The central tendency error— rating subjects toward the middle 

of the 8c»l«. This often occurs when the observer is unfamiliar 
with or uncertain about what is being rated. ^ 

The Hawthorne Effect * 

In ^n industrial efficiency study performed at the Hawthorne plant 
of Western Electric in Chicago during the 192,0' s it was observed that to 
single out a group of workers for a special research project makes them 
feel and act differently comparedio regular 'workers. The effect of ^t his 
was to bring ^bout a consistent incr'ease in productivity in spite of ' 
changes in the working conditions intended to both increase and decrease 
efficiency. Explanations for this effect point to the factors of: 
(1) novelty; (2) awareness that one is a participant in an experiment; 
(3) a modified environment involving observers, special procedures, and 
new patterns of social interaction; and (A) knowledge of results in the 
form of daily productivity figures and other feedback, ordinarily not 



I 

.i 
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syscematlcally available. 



The Experimenter Bias Effect or "Self-Fulfilling Prophecy'* In Evaluation 

Well documented In the literature, this is the bias an evaluatbr is 
inclined to project into his methodology and treatment that subtly shapes 
the data in the direction of his foregone conclusions. What the evalua- 
tor ''expects to see,'* where he directs his attention, what he ignores or 
forgets, what he remembers or records, and even the way he Interacts with 
subjects to alter their own expectations and motivational states all 
can influence the results to fit his preconceptions. 

Advantages of S<^lf Evaluation 

a. Responses, if truthful, may be readily interpreted, and inferences 
are not necessary • 

b. Sample can be small and still capture what participant? believe 
and value. 

c. Responses to such measures may be supplied anonymously; thus 
"valid feelings" may be anticipated, (in a small vocational 
progtam it may be difficult to retain an<^nymlty) 

Diaadvantages of Self Evaluation 

a. The results will almost 8urely*carry a subjective bias. 

b. The data are naturally affectively oriented, and thus difficult 
to verify. 

c. Data often are gathered by, oral reporting and may be subject to 
misinterpretatloi;!. \ . ^ ' ' 



A State ^laxi 

• 

A new plan presently operating in one state for overall evaluation , 
will depart somewhat from past procedures but will still rely heavily 
on the descriptive and self-study aspects. Since the new 1^ (VEA 1976) 
provides that all programs must be evaluated each five years, there 
*auat be some activit:' going on at aM times in at lea^t one fifth of the 
school districts. The plan lysing used consists of the following system - 
on a four y«*ar cycie, since the plan was one year late in being implemented 
and will carry through, until a new appropriation is made * for example: 

Self Assessment V , Financial Audit 

Validation •.^^^.^^^^ ^0^^ 

One fourth 
^ of districts Monitor and 

. Follow Up — each year Compliance Review 

^ ' ^. 

A copy ot* a proposed useful instrument follow^, for the self assess- 
ment and subsequent site valldntion. This rcictionnaire is most directly 
concerned with a 'tiubjective judgement of i^idividual vocational education 
programs and thut has its llmitatians. It may be useful in providing local 
•valuators with ide.as for developing their own surveys to meet statc^^^ 
^requirements. ' ^ 



35 



■ ■ I ' - 




m 


4 

f 




seat* Department of Education 
Vocational Education Self-Assessnent 

Croup I (1979-1980) ' ^ 
Form 

) 




r ^ 

Return two copies > of Form A (School) 
and two copies of Form B (Program) 
for each Vocational Education Program 
mal lined by the school to the 
appropriate regional office by 
November 10, 1979* 


LEGAL NAME OF LOCAL EDUCATIONAL AGENCY (LEa) 


County 


District 


School^ 


NAME OF SCHOOL 




Refer to the 


Public School 


1 




Directory fox CDS Code Numbers 

\ 


^ a( ^ , Certification 

I. hereby certify^ that, to the beat of my kncwledge and beliefs the information contained in thia \ 
school report ie correct and complete. 

-:- i. 


SIGNATURE OF DISTRICT SUPERINTENDENT 


Date 


SIGNATURE OF SCHOOL PRINCIPAL 


Date 
# 


SIGNATURE OF PERSON COMPLETING FORM A (SCHOOL) 


Position or title 


Telephone 

( ) 


Date 


f , . « • 


i- ■ 

» • 

ERJC , 











DIRECTIONS: Ont Forn A Vocational Education Stlf-Aaaearaant (aehool) will bar aubmittad by aach achool or othar 

organisation having Vocational Education Prograaa. Pleaaa aupply tha daacriptiva information raquaatad, 

<■ • ■ * 

Forn B ahould ba diatributad to all prograna deaignated. Vhan tha Form B'a have been completed they 
ahould be attached to tha Form A and returned to the deaignated regional vocational education office. 



FORM 

Vocational Education 



1. Plaaaa make any nacaaaary corractiona to achool 
addraaa label (including achool coda number). 
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w 2. Type of enrollment i 

Gradea 7 4 8 

Gradea 9-12 

Adult 

3. Total number of atudanta that will ba 
aerved thia year in your achool? 

Total Enrollment 



A. Does your achool have a apecialisad 

vocational guidance program for atudantu? 

Yea No 

5a Total number of guidance personnel: 



6a Total guldanc^personnel with pupil 
personnel credentials: 



A saiooL 

Self-Asaeaamant tSchool) 

7. How many of the credentialed guidance 
paraonnel have apecif ic vocational 
guidance preparation? 



8. 



9. 



Doea your achool provide job placement aarvicaa 
to vocational students? 

Yea No . 

If Yea I check the type of placement aarvica 
available: 

Cooperative program between achool and 

atate employment aervice 

Vocational teachera make referrala 

Counaelor aaaiatance ' | 

other (apecify) 

r 

If anawer ia No; are auch aarvicaa provided 
via a centralized diatrict office? 



Yea 



No 



Doea your achool have an annual and long range 
plan for vocational education? . 



Yea 



No 



If answor is No; does auch a plan exiat at (he 
diatrict level? 



Yea 



No 



4'} 



FORM B PROGRAM SELF-ASSESSMENT 



School Codo 

Instructional Program Title 



Date _/ / 

Mbntl) Day Year 



(What do you call this program?) 

1 ' 



Vocational Program Area 



(Agrltulturei Bualnesai Home Eq,| Trade and Industrlali etc*) 



00 



Vocational Program Coda 

Person Completing 'this form 
Telephone ( ) ' 



DIRECTIONS: 



This form should be completed by an Instructor who teaches In the program. It Is expected thatl the Instructor 
selected will teach one of the principal skill acquisition courses. The form should be completed In not more 
that 30 minutes. 

When the form Is completed, It should be returned to the school official who has completed Form A to be < 
packaged and forwarded to the appropriate regional vocational education office. 
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X. GENERAL PROGIUM DESCRIPTORS 



1* Uv«l of training (check pna only) 
Grades 7 and ^ 
Grades 9-12 
Adult ^ 
Other 

2. Is your Instructional program composed of: 

A series of separate 

.specialized courses? 

A single course? 

Other " . ' ■ 

3s Vhen Is this Instructional program offered? 

Day only 

, Night only 

Both day and night 

4s Does your Instructional program operate at 
times other than during the regular school 
year (September through June)? i 

\ 

5s What Is the duration of tlme» In veeH^ 
required for a student to complete th^s 
Instructional program? / 

i 

Number of weeks | 

# 

6s How many clock hours of classroom training 
ot Instruction are required for a student to 
complete this Instructional program?; 

® "4 S = . 

EMC. Clock hours . I 



7. Do you have an operational procedure for Yes Nq 
determlng whether sex fairness prevails 
in this vocational education program? 

If yes, Is there evidence* In terms of 

alteration of historical sex proportion 

trends or other evidence that Indicate 

that present procedures are effective? 



I^.- * /AILABILITY OF INFORMATION FOR PLANNING 

8s Do you regularly receive and use Information 
about the characteristics (status In school, 
test Information) of present students? 

Receive 

Use 

9s Do you regularly receive and use Information 
about the characteristics or potential 
students (from feeder schools, drop outs, 
etcs)? 

Receive _ 

Use ' 

IDs Do you regularly receive and use Information 
about the labor market? 

Receive 

, Use 

lis Do you have access to Information about your 

teaching specialty on a regular basis? 

12s Do program Instructors have the opportunity to 
.interact directly with your vocational educa- ^ ^ 
tion advisory commltte4? ' *^ 



13. Do you hav« current and projected skill 
requlrenents for the programs you teach? 

14e Are you regularly provided with Information 
about the Identity of and the special needs 
of dl8ad>j;^ntaged students? 

Identity 

Needs 

15e Are you regularly provided with infomatlbn 
about the Identity of and the special needs 
of the handicapped? 

Identity 

Needs 



* III. PLANNING PROCESSES , 
o ^ 

16. Do you actively participate In planning the 
content of proposed courses? 

17. Do you actively participate In the selection 
of courses that will constitute a program 
sequence? 

18. Do you regularly participate In overall 
vocational education planning for your 
school and district? 

19. Do you participate directly In the budget 
process that supplies resourirt for your 
programs? 

20« Indicate the year In which the last analysis 
was done to determine the occupational com- 
petencies that need to be taught. 

This year 
, Last yenr 

) 0 Two, years ago 

More than two year ago 
Unknown 




21. Based on the above analysis have written 
student performance objectives been 
developed for this Instructional program? 

22. Indicate the year In which the next analysis 
will be done to update the occupational 
competencies that need to ^e taught. 

This year 
Next year 

Within the next two years 
More than two years from now 
Unknown 

23. Indicate the year In which manpower data 
was last secured to determine the extent of 
potential employment opportunities 
available. to program termlnees. 

This year 
Last year 
^ Two years ago 

More than two ye^rs ago 
Unknown 

\ * 

24. Indicate when It Is planned to next secure 
manpower Information to revise estimates 
of potential employment opportunities for 
program tei^lnees. 



This year 
Next year 

Within the next two years 
Mote than two years from now 
Unjcnown 



i 

25. Does your Inst rue t/lonal program have an 
advls6r^ committee? 



« 



« 

26. What !• tha compotltion of your advisory 
eoBttlCtae tov this instructional program? 

Chack tha appropriata boxea 

Buainass-industry rapresantatives 

Labor union raprasantatives 

Faranta and/or studants 

■ 

School personnel 

Other (specify) ____________ 

IV. PROHOnON 

27. Do you share In the responsibility for 
communicating vocational education plans; 

I 

To your school 

To school district adminiatrators 

To the geheral public 

28« Doaa anyone actively I'^cruit students for 
your programs? 

^29. Does anyone encourage disadvantaged and 
handicapped students to enroll in your 
programs? *^ 

30e Do your program instructors have difficulty 
in maintaining a cooperative relationship 
with counselors? 

v. MANAGEMENT OF INSTRUCTION 



31e Do you furnish papers certifying per- 
formance of successful program completers? 



32. Do you use a written » approved curriculum 
guide? ^ ' _ 

33. Do you use performance objectives as a basis 
for instruction? 

34. Do you help to provide your students with 
organized subject-matter oriented club 
activities as a part of their instuctional 
program? 

35. Do you provide learning alternatives in keeping 
with Individual .needis 9 abilities » and pre- 

. ferences? _ 

36. Do you encourage studentA to consult with you 
when selecting their personal educational 
objectives? 

37. Do you assist students in frequent assessment 
of their own acheivement? 

38. Do you provide students with periodic progress 
reports? 

39. Does the school provide you with time to 
coordinate and supervise students in coopera- 
tive vocational education or vocational work 
experience education? 

40. Does the school provide you with time to 
attend advisory committee meetings? ^ 

41* Does the school provide you with time to 

sponsor subject-matter related youth organ- 
izations? • 

42. Does the school provide you with time to 
attend professional meetings and workshops 
that will enable you to maintain or enhance 
teaching skills? ^ 

43e Does the schoc^ provide you with time to 

visit appropriate business or Industrial oper- 
ations to' make placement contacts and to learn 
currant akill raquiramanta? 



AA« DOM^the school provide you with time to plan 

curriculum and Instructional materlalB? ^ 

U5. Has the school provided you with standards 
for student behavior? 

46. Has the school provided you with standards 
for student performance? 

V 

A7« Does this Instructional program provide 
students with pyt-pf-class laboratory 
experience? 

^ Vtote type(8) 

Cooperative 'vocational ed. , . 

Vocational* work expi^rlence 

* f t Community classroom 

Other (specify) * 



48.. Are there specified course prerequisites 

/or admission to this .Instructional program? 

49. Are there grade level requirements for 
admission to this Instructional program? 

Specify grade level j 



50. Are there age requirements for admission 
to this Initiuctlonal program? 

Specify age requirements 



51, Are there specified student achievement 
levels required for admission to this 
instructional program? 

Note type(s) ^ 

O ^ ) Special test • 

ERJC Cjrade point average 

Pther (specify) ' 



52. Are there specified student achievement levels 
required to remain in this InstrMctlonal pro- 
gram? _ 

Note type(s) 

" Competency test _ 

Grade point average _ 

Other (specify) _ 



.53. Is specialized remedial instruction made 
available to vocational students in this 
program who have educational deficlences? 

54. Is additional specialized vocational 

instruction made available to vocational 
students in this program who have educational 
deficlences? 

.SS« Check one or more items which best describe 
the source of the ciii.riculum materials used 
in this instructional program: 

Original or adapted by local 
staff 

Standard state curriculum 
guide 

Commercially prepared 
materials 

c 

Borrowed from similar 
pro^jram 

Developed by special agency 
or association 
(specify) 



Other <speclfy) 



36. Ar« thtre «ny voe«tlon«l studtnts from non- 
profit private schools enrolled In this 
Instructional program? 

Number of students 



•VI.. PROGRAM UTILIZATION DATA 

57. What Is the potential enrollment In this Number of 
Instructional program? (Student capacity Students 
times the number of sections) 

9th and 10th 

11th and 12th " 

Adult 

y 

Other (specify) 

^ Were students denied training this year due 

to limited facilities or capability? - 

9th and 10th * 

11th and 12th 

Adult 

Other (specify) 

VII. PLACEMENT 

58. Do you regularly provide occupational 
information to your students? ; 

59; Do you regularly provide occupational 

Information to counselors? 

60. Do you utilize your advisory committees 

tor help with placement? _] 
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61^^ Indicate the type(8) of structured placement 

services provided by^ the school which this program 
uses: ^ 

Cooperative program between school and 
state employment service 

^ Vocational teacher referral 

Counselor assistance ^ 

Other (specify) ■ 



VIII* FACILITIES 

62. Are you provided with the following In sufficient 
quantity and quality for conduc^pg an effective 
Instructional program? 

Spaces . 

Materials 

Equipment 

63. Do you participate in the selection of 
instructional equipment and materials? 

64. Do your classrooms have adequate: 

Heat 

Light 

"i ■ 

Ventilation 

65. Do you have adequate storage fo^r security of 
materials and equipment? 

66. Is your equipment typical of that used in 
industry? 
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CASE STUDY NO* 3: FOLLOW-UP 



*Tbe follow-up study has been a vital part of voca^tlonal education 
•valuation for several years. ' It is mandated through federal legislation 
and is carried 9n in one form or another in each* of the 50 tftates. The 
techniques described here most particularly relate to the reporting require- 
ments as they have been exercised in one State as seen from the local school 
district level*. The district must collect certain information to meet require- 
ments and be in compliance. Beyond that, it is possible for ohe LEA to construct 
a questionnaire that will' provide additional information of value to local 
program planners and decision makers. This additional information can combine 
both process .and product factors that are helpful in program improvement* 

' • y'' 

The Morrison Unified School District, recognizing this fact, conaJ^rUc- 
ted a questionnaire and telephone technique that has produced a very high 
and reliable response rate. Several evaluation schemes are illustrated by 
the Morrison experience, although follow-up per se is the only one discussed 
here* 

Technique 4; Follow-Up 

As stated, the follow-up process is a required activity in vocational 
education for all students who have completed a program and left school. 
In a sense' this is strictly a compliance-type activity. The follow-up (in 
the fall, for the previous June's program leavers) is required; therefore 
it ia done by districts accepting VEA funds. This is not to imply the 
follow-up system is not good; it is simply poorly done in many cases and is 
excessively shallow because of its use only for a one^year period. A truly 
valid follow-up should extend over a several year period and should incor- 
porate both vocational students as well as non-vocational students. But, 
alas - this will require the time of a graduate student who has the time 
and complete diligence to prepare a five year doctoral Research - or iji 
will require the time and talents of a^highly motivated researcher to devote 
his f fforts to the detailed task of tracking students, over a five year 
period. In eit'her case, it seems the opportunity for truly significant 
results in this regard may be slow in coming. 

However, since follow-up, as an evaluation technique, is in fact 
done each year, it behooves iis to review a few of the practices involved. 
Th^ first of these principles is "follow-up" itself. Is it a good technique » 
and may valid results be derived from it? The answer is yes - but we must 
assume certain qualifications for this statement. It must be done carefully 
and the questionnaire must be created in a way it will be easy to understand 
and also elicit the required information. A "one shot" questionnai^'e mailed 
to program graduates will probably generate no more than a 10% to 25% response 
at best. This certainly will not give us a true picture of the effects of 
vocational training* The small percentage who do respond to such a request 
may be extnemely skewed or biased. That is, perhaps 'certain ethnic groups 
or socioeconomic groups respond more readily than others, or perhaps the 
employed individual replies more frequently than the unemployed person* The 
possibilities for bias are many. A preferred technique is probably a combination 
of a telephonic survey, coupled with a questionnaire mail-out. 
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the Morrison dlsrrlct, after several years of disappointing results, 
devel^oped such a nethod. A questior laire was prepared which would supply all 
'^formation required by the state as well as additionalinfonnation required 
\district for a part of its own program evaluation. 
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Characteristics of a Good Questionnaire 



•)\ It deals with a significant topic, one the respondent 
• \%rlll recognise as important enough to warrant spending 
Ms time on. The significance should be clearly and 
'Carefully stated on the questionnaire or in t^e letter 
that accompanies it. 

b) It seeks only that information which cannot be obtained 
from other sources such as school reports or census data* 

c) It is" as short as possible, only long enough to get the 
essential data. Long questionnaires frequently find 
their way into the vastebasket. 

A) It is attractive In appearance, neatly arranged, and 
clearly duplicated or printed. 

e) , Directions are clear and complete, important terms are - 

defined, each question deals with a single idear, all 
questions aife worded as simply and as clearly as 
> possible, and the categories provide an opportunity for 
easyt accurate, and unambiguous responses. 

f) The questions are objective, with no leading suggestion 
as to the responses desired. 

g) It is easy. to tabulate and interpret. It is advisable , 
to precohstruct a tabulation sheet, anticipating how 

the dat^ will be tabulated and interpreted, before the 
final form of the question*is decided upon. This working 
backward from a visualization of the final analysis of 
data is an important step in avoiding ambiguity in / 
questionnaire form. 

The Morrison technique called for all students who had completed 
programs, to be telephoned - at night, when they were most likely to be 
home. The interviewer asked the questions over the telephone and recorded 
them on the questlonaire forms. Three tries were made to contact studeil|s 
in this manner. After that time, if a successful contact was not made, 
a 'questionnaire was maile In two more weeks. If no response was received, 
a second mall-out was pet^ormed. The result has been a 75% response rate 
for each of the two years the system was used (research has shown that a 
response rate of less than 70% is probably not reliable). Certainly the 
procedure Is time consuming and is fairly costly; especially considering 
the computerized procdsslnR of final data. In such a case, an educational 
agency must weigh what It feels it should do for mere compliance versus 
what is must do for derivation of good d^ta on which to rely for piogram 
continuance,, modification, or termination. 
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Questionnaire Construction 

The etate/ through Its federal Influence, requires InfortDatlon about 
vocational program leavers. The basic Items they* requre are: 

Is the former student: 

Working full time? ' ^ 

Working part time? 

1b job related to vocational course? 

Attending school? 

Looking for a job? 

In the Armed Forces^ 

Seeking work? 

Not seeking work? 

Hot In labor force for other reasons? 

This type of Information Is useful and does provide one measure of 
program success. Each school district Is left the prerogratlve of design*' 
ing Its own questionnaire to elicit the Information required. ' Thla quest Ion- 
aalre, If mailed, must be easily^ understandable, slQce It Is Inevitable 
•omeone will misinterpret items on evien the most carefully prepared 
instrument. The example shown represents the collecitlye thinking of several 
people over a period of two weeks, and the results obtained from it appear to 
bear the fruits of careful thought and preparation ahead of the actual follow- 
iq> process. It Was prlntied on green paper (which some people feel is an 
incentive for larger responses, although this may never have been actually 
proven). Also, its general lay-out, including the heavy black ^'tracking 
arrows*' are helpful to the former student in making his responses. A simplified 
layout is also helpful to telephone interviewers, who themselves rtiy not be 
highly trained or very familiar with uhe programs upon which they are seeking 
responses. 

The Morrison staff felt that more Information than the state required 
was actually needed to measure program effectiveness. Hence other questions 
were added. For example, question 5 seeks to determine the reasons for the 
individual having taken the course in the first place. It seemed to the 
planners that much more weight should be attached to employment results if 
the student originally took the class "to prepare for a job" (item 5a). On , 
the other hand if his reason for taking the class was "to see if I liked that 
career area" (Item 5d) , it would appear as if a lower related employment count 
would not be as significant. Other questions too, were added such as numbers 
7 through 12. These questions reached beyond the original requirements and 
•ought to, bring in some Information that would be helpful in evaluating parts 
of the instructional phase of the program; that is, process evaluation versus 
product -evaluation. 

The main points to be made with regard to questionnaire construction are 
that each instrument will be carefully tailored to meet the needs of those 
who desire to obtain Information, it will be prepared for ease of administration, 
it will be prepared for ease of tabulation of results, and it will be prepared 
for eliciting the greatest possible, response. 
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Smpllng Versus Total Coverage 

A strAtlfled random sample may be used to conduct follov-up studies 
of total district vocational program effectiveness. The posture assumed In 
this manual Is that total coverage Is a better approach. The reason for 
this Is that if gives the district the capability to have a tiore complete 
fiX^ on a larger segment of its graduates; thus permitting the opportunity 
to provide employment and other assistance to them in the years after they 
leave schodir 

Adoantagee of Follow^Up: • * 

a« Can potentially get nearly total coverage of a. population. 

b. Not as jtime consuming as interviews. 

c. Gets answers to exact questions upon which one requires 
information. ^ * 

' d. A yide rimge of .people and occupational types can be 
covered* 

DiBadvantagea of Follow-Up: 

a. Responses may be In favor of one type of bias. 
b« There may be sampling errors- (If a sampling technique Is 
used). 

c. Response rate will be low unless process Is repeated 
Ind/or combined with telephone techniques. 

d. Can be costly if the full potential of Item c Is exercised. 



O 47 

ERIC 



VOCATIOfiit EDUCATION 
LONGITUOtNAL FOLLOW*UP STUDY 

Morrison Unified School District 
Ureer Education Unit 



Mn$w0r sll qu0»tion» that appiy to you by marking the appropriate Mpece and adding the informa^tion requested. 
Additional'' cotments mjv be made in the space provided on the reverse side of this questionnaire. The vocational 
elaag referred to m the questions is the one in which you were enrolled as indicated^on the label above, ifeturn 
immediately in the enclosed self ^addressed, stamped envelope. Thank you. 

1. Did you'complttt the course printed ebove? « ye8( ) no( ) 

2« If no; vhet vocetlonel course did you co«!ilete7 \ ' 

f 

tiegardless of the name of the course you completed, would you please answer the following questions? 

3. Are you ftlll attending high fchool? • •••• • yes( ) no( ) 

A. Are you in thf armed forcee^ f yet( ) no( ) 

3. Whet was your reeson for taking the course? (Mark one space only) 



e. ~ To prepars for a Job. ••• i.( ) 

b. To prepare for advanced training. ••• ^ ( ) 

c. To improve existing Job skills ( ) 

6. Are you presently employed? • yes( ) 



d. To see if I liked that career area.. ( ) 

c. To learn a skill for my personal use ( ) 

f. Other ( ) 



no( ) 



IF YES^i 




6A. Is this your first Job since leaving 

school? ; % yes( ) no( ) 

6E« Is your Job: 

«. Full-tine ( ) 

b. Fart*time «....( ) 

6C. How closely i^ your job related to 
your vocational course? 

a. Very closely ( ) 

b. Somewhat i •••( ) 

c. Not related .^....v ( ) 

6D. Flease print your Job title here: 



IF NO 

▼ ■ 

6AA. Have you been employed since 

completing the course? yes( ) no( 



6BB. What is your present situation? 

a. Looking for a' Job.. ( ) 

b. Not looking for a Job ..v*^ ) 

V 

c. Attending school full-time. ( ) 

d. Attending school part-time....;....; ( ) 



6CC. If you are in school, is your course 
of study related <o the-course which 
you took? yes( ) no( ) 



(For example: 3ales Clerk) 

■< 

6E« Are you also attending achoolT 

«• Attending full*timc ( ) 

b. Attending part-tine ( ) 

c. Not attending school ; ( ) 

6F. If you are in school is your course 
of study related to the course which 
you took? ye8( ) no( ) 

7. Did the vocational course which you took provide an understanding of future careers in that area?.yes( ) no( ) 

8. Do you feel you received an employable skill from the vocational course? * yes( ) no( ) 

9. Did the course offer instruction on how to look for and apply for a Job (in terms of applications, . 
Interview techniqut^H, and sourccsi of employment information)? yes( ) no( ) 

10. Did the course offer tniitruction on how to keep a Job (such as, being on time, using appropriate dress 

ar i grooming, and r.«?ttln>; along with people)? yes( ) no( ) 



11. 



Was placement service offered to you? yes( ) no( ) 



12. If you were asked to assien an overall "grade'* to your vocational course of instruction, what would it be? 

V/ b ( ) 



A ( ) 



C ( ) 



D ( ) 



10/76 
Form 43 



F ( ) 
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CASE STUDY NO. A: PROJECT LINKAGE ' 

J 



High school students with learning disabilities (educationally , 
handicapped) have traditionally been taught, in accordance with state 
lav, in classes limited to an enrollmi^nt of 12 to 15. This ;Ls the opposite 
of ao^called *'mainstreantlng,** where the students are placed in regular 
claases with peers of more equal abilities. Either of these two methods 
of instruction has certain inherent problems. In the first instance, the 
aamll classes are often placed in a school campus location that is not 
physically desirable. In any case, wherever the class is located on a 
regular campus, the handicapped atudents may feel a sense of isolation and 
are often referred to as ''different** by the rest of the student body. In 
tha second inst^ce, /'mainstreaming" can cause serious problems with learning 
dlfeability-type Youngsters because it is difficult for them to keep up with ' 
the atiffer competition from regular students. 

' The Claybume Unified -School District, in an effort to provide an alter- 
native format, prepared an application for funding under Title IV-C of the 
BSEA Act. The funding was approved, and the program commenced operation In 
July 1977. The two principal things that set this program apart from other 
such classes were: 

1. It was to be taught in a location removed from a conventional 
school setting. 

2. It was to have a car<eer exploration and vocational educational 
component. ^' 

Edgewater University, a private school located in the middle of the 
Claybume School District, volunteered the use of one of . its classrooms 
for the students* use, thus removing them from a traditional high school 
setting. 

The district paid the salary of one of its regular special education 
teachers, but his two aides, student transportation, instructional materials 
and evaluation were paid from IV-C funds. Since this was the case, certain 
tjrpes of project components were required by IV-C, and the forms used in 
the evaluation reports were rigidly structured. A fair amount of Ingenuity 
was allowed, however. In the methods of collecting and interpreting the data 
needed to measure the objectives, as well as in the final narrative report. 

The Project consisted of seven basic jcomponents^ which were: 

o Instruction (basic skills) 

o Increased self-concept 

o Work exploratory participation 

o Increased interpersonal social skills 

o Inservice training for teacher 

o Parent awareness (2 components) 

This project followed the usual state Title IV-C guidelines for pro- 
gram preparation and for ev^aluation design and reporting. The important 
evaluation considerations tor us to look at here arc: "comparison proups 
and sample size'*, as well as the requirement for standardlz d testing. 
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Technique 5: Comparison Groups 



In order to effectively observe any changes In behavior of a group 
of students, It Is first necessary to establish baseline data against which 
to make meaningful comparisons. Such data establish what the subjects were 
like before the treatments began. Common procedures for gathering baseline 
data are the pretest and the comparison group. The latter la Important 
where there Is an expected treatment effect and the absence of treatment Is 
the baseline. 

The comparison group should be matched with the test or experimental 
group in every aspect except for the treatment being offered. For' example, 
In the case of: Project Linkage, the Ideal situation would have been that 
both test and comparison groups had the following points In^common: 

1) * Twelve students In each class* 

2) - All 11th graders. ♦ 

3) * Equal amount's of boys/glrls and mlnorlty/hon-mlnorlty. 

A) ~ All students ftrtified roughly to equivalent ability levels. » 

The only difference should be» one group receives a treatment some- 
what different than the other group (test class versus comparison class). 
In the case cited, of on^y 12 students permitted by state law Ih each class ^ 
the validation of a bona fide comparison group, meeting all c/lterla, was 
difficult. However,, the principles of comparison group/test/^group analysis 
were adhered to completely and were accomplished within the^ realms of type of 
student served and relatively small "N"s to work with. S^p^e size is belief; 
dealt with in another case study (No. 6), where a scientifically developed! 

methbd for determining a valid sanqple is displayed. ' 

. ' /' I • ■ 

In a case such as Project Linkage, the usual state requirements dictated 
pre and post testing with a nonn-referenced test "to be selected by the I 
evaluator.'* This brings up one of the dilemmas facing the local evaluator; 
what test to select, and does the selected test actually measure the outc^es 
to be analyzed in relation to the projects* objectives? In a case such as 
Linkage, where the total N was so small the sample size in effect had to Wi 
the entire experimental class. The selection of a comparison group was even 
m^ve difficult. Since the law prescribed only 12 students per class, it was 
impossibly to find a single group in the city schools tliat matched the require- 
ments completely (i.e. all 11th graders, boy/girl, black/white, etc.). 
Therefore^ it was necessary to draw the comparison group from two other 
classes (in two schools) to attain a reasonable match with the test group. This 
is an atypical situation, and is not recommended from a purely research point 
of view. However, from the position of the day-to*day realities of field 
evaluation, it is illustrative of the need to often stray from a research 
posture to attain the ends of doing a reasonably creditable job of evaluation 
%rlthin the constraints of state requirements dictated by categorical funding 
sources. - y 
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Advantages of Campcan.3on Groups: 

a. Allows a good gauge of treatment effects. 

b« Reduces chance of error in collecting valid data in 

experimental, designs, 
c* Allows baseline data to be collected (when absence of . 

treatment is the baseline). 



DtBadvantagee of Campconaon Groups: 

a. Difficult to match on all variables but for the iDne t6< 

be tested. V ^ 

b. Not always possible to get a renodmlzed group. 

c. Difficult for comparison group to "experience all things" 
except fdr the critical factor. 

Requirements for Standardized Testing 

Standardized testing, using 9 nationally normed Instrument, is 
virtually a "must" on any project receiving federal or state funds. ^ This 
la true displte the fact non-standardized measures of effectiveness are 
often much more appropriate In vocational programs. Some of the qualities 
that vocational classes seek to create In students are difficult to assess 
with academic achievement £ests. Nevertheless the general public as well 
as memtiiers of the state educational hierarchy usually look at academic 
achievement as the single most Important item in any public school program. 
^Usually the evaluation guidelines of any project receiving extracurricular 
funds will require standardized testing. Eve^ if an evaluator is utilizing 
a design of his own choosing he would still be well advised to build in a 
standardlj^ed testing component. Or if this I3 not feasible he should at 
least provide some sort of comparison Qf vocational student performance 
at criterion levels with hl^ test score history. In a climate of national 
concern over declining test scores the best way to sell vocational education 
is to show it shelps (or at least doesn't impair) performance in the basic 
skills of reading and mathematics. 

Displaying Test Results 

Test results, even for such a small project, may be tabulated ani 
displayed in a variety of ways. If the gains of treatment group over 
comparison group were especially significant, it is to the advantage of 
the evaluator to show these gains as dramatically as possible. The following 
chart Illustrates the bar graph method of display which emphasized the growth 
of one group over another in easily understandable form. ^ , 

Structured Interview 

The structured interview technique was described in Case Study 1, 
although there it applied to a very broad base of interviewees. On smaller 
projects or individual programs it also is often useful to employ the 
interview. For example, in LINKAGE, one of the components, as noted, was 
"Inservlce training for the teacher." Since the Inservlce training was of 
an Informal nature it was difficult for the evaluator to determine the gains 
in teacher knowledge over time without using an interview process. In this 
case the interview form was quite simple, but it served its purpose as it 
was Intended; to provide a degree of form and structure to the talks (one 
at the bcglnnlnR of the year, one at mid-point, and one at the end). A tape 
recorder was always used. This is a valuable tool; but be sure it is used 
with the permission of the respondent. A sample structured interview form 
is shown. 



The interview gives an opportunity to measure the "practitioner 
variable"— those characteristics of teachers and instructional support 
staff which reliably indicate the status, quality, ami progress of the 
iaplementation process. It may also present an oppprtunity for a 
retrospective review to help develop a reasonably complete account of 
events leading up to the current status of a project. 
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PROJECT LlinCACE 
MEAN RAW SCfbRC AND MEAN 
CRAD^jEQUIVALENTi^ PRE-POST 
PROJECT CUSS vs. COMBINED 
CO>Q>ARISON GROUP 
1977 - -1978 ■ 



TOTAL TEST 




PROJECT COMPARISON • PROJECT* 

NOTE: Shaded area Indicates amount of f.rowth 



COMPARISON . 
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CLAYBURNE CITY SCHOOLS 
PROJECT LINKAGE 
SPRING, 1978 



Structured interview tjith project jeacher 



NOTEi One of the components of the original project related to staff 

development and Improvement of skills on* the part of the teacher. 
It was tiiltlally decided the only way to accomplish an evaluation 
of this. phase would be through a series of formal and informal inter- 
views; evaluator and teacher. The purpose of this ehd-of-year inter- 
view is to attempt to gain insights into the "Pulse of the Operation" 
over and above the hard data required in the year-end IV-C reports. 

^ c !• We were tequlred to do standardized testing for academic achieve^ 
nent and attitude development. What other things do you' see 'as 
allowing us to get a "feel" for how the project is really doing? 
Please list: e.g. drop outs, attendance; and law enforcement 
violations. 



2* Originally we had talked about using, the CTBS aho other stand- 
ardized measures. We ended up with the PIAT and KEY MAIH . Do 
you consider these good measures of achievement for your kind 
of student? 



3. Also we heeded to measure social skills; You arid your staff 
came up with the ATTITUDE INVENTORY for this purpose? Was it 
satisfactory and would you recommend it for the future? 



4. In July We agreed all of us were staftiftg at ^'prBUfid ifefb'* on 
curriculum. Please list a few ekamplos of what wie afe tiSihg now 
and state how useful you feel our materials will be for ultimate 
dissemination. 



5* What ate two or three of the major '^pfoblcmfe" 6He8UtiE6f6d ^iifing 
the year I atid how did wc solve them? 
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CASE STUDY NO. 5: BUSINESS CAREERS i 



A vocational "program" I9 usually deflnded as a series of^ courses 
(of sometimes only a single course) » bearing a slx-'dlglt U.S. Office of 
Education code number, and leading to a so called saleable skill at the 
cotopletlon of the series. It Is a sponsored activity, more often than not 
troxfi public funds; aimed at mitigating a social pr educational problem or 
improving social or economic welfare. A common characteristic is that 
. p^rpgrams are ultimately cpncerned with change. If a vocational program does 
not develop a skill as such, it at least should give a solid base €rom which 
the student can branch off to a related skill development course that will 
provide a more indepth training. Business Careers was just such a one 
semester course which was adopted in nearly all of the high schools in the 
Henderson Unified School District. 

• t> . * 

Henderson is one of the largest cities in its state, and at the time 
of the adoption of Business Careers, had twelve comprehensive high school? 
and three alternative high schools. The program was adopted in all but 
three of the 13 school total. The evaluation of this program extended over 
a three year period, but it is basically the activities of the last year 
that we shalL illustrate here. Many evaluation techniques were used to 

* bring in both"" qualitative and quantitative data from several sources. The 
two. techniques to be particularly illustrated are "Criterion Testing" and 
"Item Analysis." The comprehensive schools were representative of the city; 
running the gamut from upper middle class neighborhoods to schools with a 

^high degree of disadvantagement. The three alternative schools were the 
career education high school which is a cit3rwide model, ^nd the two 
continuation schools. There was therefore, a broad representation of students 
and schools. The state in which the Henderson District is located developed 
a common core of 18 competencies in business education and further developed 
a program guide including objectives, criterion outcome levels', tests, broad 
lesson plans, etc. The local district, through a steering committee of 
teachers, modified and adopted the guide for its own use. The 18 basic 
competencies in the common core business program were: 

Adding Machine Dictlqnary 

Data Processing Telephone Directory 

Grooming Zip Code Directoy 

Handwriting . Simple Business Records 

Instructions Sort, Check, Verify 

Job Application spelling 

Making Change Telephone 

Math Compqtations Work with People 

Oral Communication Written Communication 

Each of these competencies ^had a LAP (Learning Activity Packet) 
(propored by 3 committee of teachers. These packets allowed each individual 
Mtudent to work at his or her own pace and were designed to assist the 
Student in achieving the desired criterion level of performance. 
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Technique 6; Criterion Testj 



Tho first step In criterion testing is obviously to define the compe- 
tencies to be tested and then establish the levels of performance that are 
considered necessary for mastery of the skill. ' In our exampl.6 of 18 different 
skill areas, with a LAP and related test for each one, there will naturally 
be a number of test Items to measure the level of performance. The selection 
of test Items and level of required performance Is clearly a subjective 
Judgement on the part of the teachers preparing the materials. Ideally 
the most competent and "job-wise" teachers in the district should be selected 
for such a task. An example of crltcrlon-based competency might be: 



Condition 



> Performance 



Standard 



Given a list of 25 words 
and phrases, commonly used 
on employment forms and in 
test instructions and a 
list definitions 



the student will match 
the vords/phrases to 
the definitions 



with at least 20 
correct responses 



Therefore a criterion level of 20 correct responses out of 25 possible 
represents mastery of the skill. Criterion testing is not the same as 
standardized testing. The students are not being measured against a 
nationally normed group of youngsters. They are being tested strictly on the 
skills deemed necessary for the program, and theoretically 100% could meet the. 
criterion level, as opposed to standardized testing where the "normal" situation 
would be to have one half above and one half below. In actual practice of course. 
It is unlikely that all students will meet the criterion performance standards. 
tt it is found that too few students attain the. desired number of correct items . 
In any given test it may be well to re-examine the ci4teria and/or the test 
items for corrective measures. The ideal steps to follow in criterion testing 
would be: 
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1. State the evaluation criteria clear, observable terms. 

2. Build or select several measures of each one. 

3. Sample items from the above measures across randomly 
selected subjects! 

4. Use appropriate statistical techniques to analyze and 
evaluate outcomes. 

Advantages of Criterion Testing : 

a. Can use 'levels of items thai would not be acceptable on 
norm- referenced tests, e.g. items on which nearly everyone 
gets a correct response. 

b. Because items can be retained, irrespective of their 
contribution to response variance, the evaluator works 
with more descriptive rat^sures of vocational behavior. 

c. Tost can be dcsi,r,nod usin;; items that are specifiGally 
reflective of a given vocational program. 

Dicadvantagca of Criterion Tec ting: 

Selection of Items npproprialc to the instruction offered. 



a. 
b. 
c. 



Decldinj', on the proper criterion level to strive for.' 
Tests must ol ten be developed by people inexperienced in 
construction of Iniitrumeuts. 
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Technique 7: Item Analysis . 

In order to determine If the test Items are all *'good'' It Is sometimes ^ 
veil to perform an item analysis. lu terms of process evaluation this may 
also be- helpful to the teacher In adjusting the classroom emphasis to correct 
a perceived deficiency In one or more particular areas of Instruction. 

In the case of Business Careers the Item analysis was done very simply, 
but was adequate for the needs of program managers. The large volume of 
students In the cltyvlde program (600) precluded looking at all of th* tests. 
Therefore a random sample of students drawn from three of the schools, 
represefitlng all socioeconomic groups was used. The tbst Iteois were tabulated 

and were divided Into those which were: 

> ■ 

« Answered correctly more than 75% 
Answered correctly 50 to 75% 
Answered correctly 25 to 50% 
Answered correctly less than 25% ^ 

A somewhat more sophisticated method of Item analysis of a true-false 
or multiple choice criterion test would embrace the following steps: 

1« Rank order the distribution of test spores from high to low. 

2* Divide these ranked scores Into two contrasting groups. The 
division point Is arbitrary but common practice Is to split 
the data In half, fo'rmlng the top and bottom halves; or to 
Increase the contrast « compare upper and lower quarters. For 
i^ase of computation, use two groups of equal slz^. 

3* Construct a four-fold table based on two pairs of categories. 
; The first pair Is simply the category wrong-right (for a 
specific test Item); the second pair Is the category high 
scorers-low scorers (depending on how the ranks were divided) : 

Wrong Right 

High ' 



Low 



This method permits an analysis of test items answered in relation to 
the general overall scoring level of the individual student. That is, an item 
^nay be looked at in relation to how many high achievers or low achievers mastered 

it- 

The Business Careers program people were able to use both the criterion 
testing and the item analysis techniques for curriculum and test improvements 
"during a summer workshop following the close of the school year. The evaluator 
In this case was therefore able to exercise both process and product techniques 
to help with program improvement. 

Advantagca of Item Analycio: 
* • 
a; Permits the Improvement of student tests, 
b. May indicate wo;ikni»8scs and/or strengths in a vocational 
Instructional pro^;ram. 
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Diaadyantagea of Item Analyaia 



«. Consumes large anounts of time. 

b. Efficacy of Items for high achieving and low achieving groups 
may be misinterpreted. 




CASE STUDY NO. 6: EXEMPURY WORK PROJECT 



Every year in this country about 100,000 men and women are released from 
federal and state prisons. Statistics indicate that A0% will return within 
one ,ye'a^« Since the cost of maintaining an offender in prisoh is about 
$18,000 per year (not to mention the waste of human resources) , it seems 
apparent that there is an urgent need for a system of rehabilitation, or 
most specifically of •'diversion," that is truly effective. 

The 40Z annual recidivism rate is a figure generally accepted for adults. 
The Bayview County Juvenile Probation Department has quoted similar statistics 
for teenage youngsters in their area. But indeed, many localities experience 
an even higher rate of return to delinquent activities. For example, the 
Providence Educational Center Pr'^ject in Center City indicates that perhaps 
the recidivism rates of Juveniles placed on probat4.on could be as high as 
70X. In overall terms, it seems clear from the literature that A majority of 
the Juveniles adjudicated by the court committed further offenses. 

The Juvenile crime rate has been well publicized throughout the national 
press. It is estimated that about 25% of stranger-Qo-stranger crimes are 
accounted by youth under 16 years of age. The Clayburne Police Department 
made between 16,000 and 17, 000 Juvenile contacts in 1975. In Clayburne 
one out of every nine youngsters is expected to. appear before a Juvenile 
Judge before he is 18. 

In response to these validated conditions the Exemplary Work Program was 
started during July 1975. The research and demonstration aspects of the 
project were to be emphasized, although a direct relationship to the 
criminal justice system was a provision for the release of funds. The basic 
hypothesis, to be examined from a research point of view, was that paid 
employment on meaningful jobs can be instrumental in diverting juveniles 
from criminal activities. In this sense it differed from most previous 
federal projects where education and resocialization have been the prime 
points of emphasis. 

The Clayburne Unified' School District received a grant of $900,000 
(to be matched with $100,000 of local funds) from the Law Enforcement 
Assistance Administration to develop and pilot a paid work situation for 
about 500 potential juvenile offenders; also including quite a few 
Juveniles who had known prior offenses. 

The evaluation design for the 10-month project was left up to the 
district within certain parameters established by the Law Enforcement 
Assistance Administration. The design initially consisted of a variety, 
of process measures f or ,the purpose of determining (a) what wektu into the 
program, and (b) what happened in the program. The major part of these 
data were collected by questionnaires and intervi^s with direct project 
partlcipaifts, i.e. students and project staff and by similar techniques 
%rith ind irec t participants, i.e. parents, employers, and general public. 
Case studies on selected project participants were also considered a 
part of the process phase of the data collection. Some of the question-' 
naires used in this phase of the evaluation arc included for the purpose 
or providing the reader ideas for creating questionnaires of his own. No 
Standardized or commercial measures were used in this pro^;ram evaluation; 
all questionnaires and data collection rofjtcrs were designed by the assi£ned 
evaluator to fit the perceived needs of the project. 
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The second part of the evaluation, and the emphasis that was demanded 
by the funding agency, was "hypothesis testing". This aspect also created 
the greatest challenge to the Innovative and creative talents of the eval* 
uator. No guidelines or previous work was available upon which to draw 
ideas from the past. The hypotheses to be tested were: 

1. Measure recidivism rate of 60 project part:'.clpants who are 
known to have *'prlors'\ the recidivism rate of known priors 
will be reduced by 12.5% (to SSZ) from the known 40% rate 
existing In Bayvlew County. 

2. Measure **lnltlal contact** of 120 project participants who 
are believed to be "non-prlors." Rate of non-prior contacts 
will be reduced 12.5% (to 10.5%) from the calculated 12% 
probal^lllty now existing In the police beat/zip code areas 
served by the program. , 

Technique 8: Hypothesis Testing and Sample Size 

The two hypotheses Just outlined are clearly written In quantifiable 
and measurable terms. The first step in hypothesis testing is to arrive 
at a reasonable hypothesis of some event occuring or not occurlng (null 
hypothesis). In the two hypotheses cited the parameters were arrived at 
as follows; (1) the County Probation Department showed the recidivism 
rate of Juveniles to be 40%. (Recidivism is defined as a person being 
arrested for the same type of crime within a one year period after an 
initial arrest or release from a detentloti facility). Since 40% was a 
known figure, experts in the field of Juvenile' criminology were consulted • 
to establish a reasonable reduction rate by project intervention. This 
introduces the basic evaluation principle of obtaining '^expert Validation" 
of a proposed technique. (2) The pro^blllty of initial police contact of 
non priors was calculated by using ?oace Department, Juvenile Division, 
arrest data and comparing it with youth population in the districts 
served by Exemplary Work. This probability was' calculated as 12% in any 
given year. In order to make the probability projection' as accurate as 
possible, three year trend data were used and averaged to arrive at the 12%. 
Again, consulting juvenile crime experts, a 12.5% reduction figure seemed 
reasonable and capable of measurement. 

The statistical measurement of these hypotheses was difficult at best. 
However some positive results wete noted, especially in item (2). This 
presen^ts a question which must always be kept in mind in educational evaluation. 

Statistical Signifiaanae versus Practical Significance 

In the researcher's zeal to obtain statistically significant 
findings, he often overlooks the more relevant question: Is 
it educationally significant? Is the difference large enough 
to be practical? Are the gains Important enough to be Wortli 
the cost and effort to obtain them? This question gets at 
the trade off factor in educational decision-making and Involves 
the problem of accountability in cost effectiveness ecomonlcs. 
Even when these practical matters are settled or in the background, 
there are vnltiablc conBldcrations of social and psychological 
nature that frequently override a choice based solely on statis- 
tical signif icancc-, 
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Coaparlson Groups and Sample Size 



The hypothesis testing, as described, obviously dictated the need 
for comparison groups that Is some body of youngsters who did not receive 
whatever treatment the project provided. In the case study ve are review- 
ing here the size of test and comparison groups (60 and 120) was more or 
less arbitrarily arrived at by Intelligent discussions and understanding 
of what might be realistic In terms of time and personnel constraints. 
Comparison groups should be matched to test groups as closely as possible. 
That Is, If the test group Is made up of 30 fifteen year olds 'and 30 
sixteen year olds; AO boys and 10 girls; and 15 minorities, then the 
comparison group desirably should match the project group In all aspects - 
except for the treatment Introduced. In this case the "treatment" was 
paid employment. 

While the sample sizes In this case were not as ^precisely determined 
as they should have been on a sound research basis they were determined 
on a "what Is practical basis." The following table shows how on^ may 
determine sample size In a more scientifically oriented mode. 



HOW LARGE MUST A SAMPLE BE? 



RANDOM SAMPLE 


SIZE 


CHAkl 


(.10 LEVEL OF CONFIDENCE) 




For a Population 
of N 


P of \ 
Cases 


Take Every 
Nf.h Case 


Percent 
of Total 




10 


10 




all 


100% 




100 


50* 




\ every other 
\ case 


50% 




150 


50* 




\ every third 
case 


33 1/3% 




200 


50 




\ every fourth 
case 


25% 




500 


60 




every eighth 
case 


12% 




f 

600 


60 




every tenth 
case 


10% 




750 


61 




every twelfth 
case 


8% 




1000 


63 




every fifteenth 
case 


6% 




1500 


64 




every twenty- 
fourth case 


A% 




2000 


' 65 




every thirtieth 
case 


3% 





*Actually the number calculated by formula is less than 50 but for convenience 
of choosing it is recommended that one round up to 50. 
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Advantages of Hypothesis Testing: 

a. Permits measurement of a pre-coticelved concept. 

b. Permits use of comparison groups. 

c. . Permits statement of researcher's expectations In 

positive terms* 

Dieadvantagea of Hypothesis Testing: 

a. There may be other plausable hypotheses than the oneCs) 
selected. 

b. Difficult to establish proper levels of significance without 
prior experience In the study area. 

c. Possibility of Introducing error In acceptance or non-acceptance 
of a null hypothesis. ' 

Technique 9; Case Study 

Although some writers contend that the '^one-shot case study^' hardly 
qualifies as a respectable data-collection design* there are Instances 
when evaluators might employ It.' Essentially « the design Involves the 
administration of measuring devices to a group of learners who have ^ 
received some sort of educational treatment. Schematically, the design 
looks like this: 

7' ^- M ^ . 

One-Shot Case Study 

4 

The design Is referred to as a case study because it more closely 
resembles a clinical case study than anything else. The evaluator merely 
notes what happens to learners who have received the Instructional treat- 
ment being evaluated, and previous student background factors are consi- 
dered. 

In this Instance the case studies were only a peripheral part of 
the total evaluation aad were not performed In depth. The full c^se study 
technique Itself would be sufficient to consume the major portion of an 
evaluation manual. One of the sample ca^e studies from Exemplary Work Is 
shown. The data gathering form used by the evaluator Is In the collection 
pf forms Which follows. This form was used to obtain Information from peopl 
who. had had contact with the subject, especially project counselors. 

CASE STUDY 

LQna, age 16, entered her first EVP counseling session 
hostile about school, but shy about discussing the problem with 
the counselor. She had previously been exempted fvm Jackson 
Junior High School for truancy. Her home life was not very 
cohesive nor did she haoe interested parents. Land's mother 
had been ineffective in controllirvj her children and did not 
provide a seme of togetherness in the family. Once Lana was 
placed in a job, she reluctantly gave a portion of her salary 
to her fanily to cupplcmcni welfare benefits. The counselor 
detected that Lana resented donating a portion of her salary 
to her unemployed mother and ^^lazy^^ brothers afid sisters. 
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In job placement oaunaeling^ Lana expressed no parti^ 
eular interest but though she would enjoy working with children. 
She was placed in a day care nursery as an aide, however, she 
was unhappy with this assignment because she did not feel use^ 
ful. She was removed from the job and placed in a ^^Hold pattern*^ 
for additional counseling. During the interim, she worl<3ad 
temporarily in the center ^s office assisting the secretaries 
and receptionist. It appeared to the counselor that Lana 
became more relaxed and s^^^confident aa she interacted with 
other co-workers her own age. She was then placed in a teacher ^s 
qide position at "the Southeast Involvement Program. Lana was 
already attending classes there, and the counselor felt this 
assignment would present a positive side of school. 

During the interview with her teacher /supervisor, she 
oamented on Lana^s low self-esteem and her messy appearance. 
As Larjifl began to experience success in her classwork, receiving 
praise from the teacher, and recognition- by - other classmates i 
she cautiously gained confidence and\egan to improve her ap^ear^ 
once. In the small group classes, I/m&estgblished an identvtu 
with a group and felt a sense of belonging. Besides the group 
affliation, Lana^s work brought her addi^ipnal attention by her 
peers. As she earned a salcay, Lana was aBle to buy herself 
clothes. Her teacher/supervisor noticed a vaqt improvement once 
Lana began outfitting herself. Lana placed a great deal of 
importance on clothes; the new clothes helped boost her self^ 
image and gain further approval." As .a Worker, Lana became 
diligent cmd wanted to be ^'kept busy.^^ She enjoyed task-^ 
oriented duties. She indicated that she became interested in 
a career as a computer key punch operator. 

Lana expressed her happiness with the program as a job 
placement and individualized personal counseling service. 
She felt there were too many tempting escapes, such as truancy, 
to help avoid attending comprehensive schools. Until her 
involvement with . Ex&rrplary Work, she was avoiding school and 
wasting her time. Her work experience and the Southeast 
Involvement Program helped her use her time constructively. 
She was able to trust her counselor and teacher to the point 
of seeking them out for personal help, something she was unable 
to do with her mother. Lana felt that the attention of these . 
people motivated her change. 

Adtiantange of Case Studies: 

a. Can give an overall picture of a project's effect on one or more 
individuals and may shed insights on the program's impact. 

■ ^ 

Disadvantage of Case Studies: ^ 
Si. Interpretations may be overly subjective. 



/ 
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ERIC 



63 

\ 7, 



Ewp • ; 

CASE STUDV OUTLINE 

Give details and explanaclons of the Items listed below as they apply to the student's 
background. Mark *'no Infomatlon*' or "not applicable" for appropriate sections. We 
are Interested In patterns of change shown by the student In any of these areas since 
his entrance in the program. For exaaple, a student's crlninal record may have 
Improved since his contact with the program. We are not trying t,o show, that the 
program is solely responsible for this Improvement but is a contributing variable. 
You may attach additional pages to further elaborate on any items or provide supple- 
mentary information you. feci is valuable. 

I, GENERAL INFORMATION: 

/ 

Kame . Date of Birth [ 

Place of Birth [ ^| 

• Address - [ 

— — - . : j ^ 

Name of Parents or Guardian (Father) / • 



(Mother) 



(Guardian) 



Occupation of Parents (Father) 



(Mother) 



(Guardian) 



Nationality of Parents (Ethnic code may be used) 

(Father) 



(Mother) 



(Guardian) 



11. TEST DATA: 
Achievement 
Aptitude 



GPA may substitute 



Personality and Temperament 

/ 
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III. HEALTH: 

Phyalcal 

Mental 

IV. SCHOOL HISTORY: 

Scholarship Record/award (a) — (If no outstanding awards compare the student's Jr. 
High academic record, CPA, attendance, etc., to his. present school record or 
oost recent attendance) . 



Types of Schools Attended— ( explain student's activities within types of schools 
and reasons for changes; for example: Public Schools— attended 3 Jr. Highs because 
of disciplinary transfers, or parental ethnic transfer requests, etc.) 



Changes from School to School— Why? ( EX. truancy, military family, may be an 
extension of the above reasons) 



V. HOME CONDITIONS: 

Social and economic status ( as It applies to the student i.e. supporting family, 
welfare recipient, etc) 



Ccneral Atnoaphere of the Hone 



(If known, consider siblings, parent-child relationship, grandparent relation- 
ship or any other Inidlvldual relationships with the student.) 

?upll*s Attitude Toward His Hone fif living on own why and Is there still 
contact with hone ?) . 



VI. SOCIAL RELATIONS: 

Conpaulons and Friends (Gong Involvement, a loner, etc.) 



Relations with Teachers (May reflect student's attitude toward school.) 
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Out-of-School Activities 



VII. INTERESTS: 

Educational and Vocational Plans 



Interest/Hobbles at Tills Tloe-CHave Interests changed since your contact with 
students? How?) ■ . 



Curriculum In which enrolled-^xample: EWP classes,, ROP classes, regular day 
school, adult school, working towards what kind of a degree) 



'1 
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CLAYBURUE CITY SCHOOLS 
EXEMPLARY WORK PROJECT 

STUDENT EVALUATION - 1975/76 SCHOOL YEAR 



Name of Interviewer 



Location of Interview 



Date 



Your Grade 



1, Your Name 



2. Your Work Location (Employer) 



3, How long have you been in the project? 



A, Have you received what you expected from this project when you went into it? 



Yes 



No 



tf. Please describe how you feel about your job experience up to this tine. 



6. Have you changed jobs since being on the project? 
If yes - why have you changed? 



Yes 



No 



?♦ School you attend now 



8« Please describe* how you feel about your school experience up to this time. 



9. in the blank provided at the beeinnlnc of each of the three statements below, place 
a number that best expresses how you feel about It. Use the following 1-5 scale?* 

5 - Fantastic! 

A - Gredt * • 

3 - O.K. 

2 <- liombed 
1 - RlpofiT 

A. How much have you enjoyed your present job. 

B. How wpuld you rate your classes (English, math, etc.) - at the school 
T" you now attend. 

• c. Kate your experiences in this project into your total life. 

(Is it one of the most important parts of the life you lead now?) 

10. If you could change any parj;. of this project what would it be?^ \ 



U. What do you like best about it?_ 



12. Has the staff (counselors and teachers) been helpful to you? 
\ Yes No undecided 
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1975 - 76 

CLAYBURNE CITY SCHOOLS 
EXEMPLARY WORK PROJECT 

STAFF SURVEY 

(To be cpmpXeted by all staff 
attached to the project) 



Date 



Name of Interviewer 



Title 



Staff Members Name (optional) 

DIRECTIONS: The first year's evaluation is concerned with the implementation 
and the process of establishing the project; the program staff are to be held 
accountable for the process. Assist us by providing candid responses to the 
questions below. In the first eight questions please circle your response on 
the scale of 5 to 1. In the last two questions please express yourself in 
narrative form. THANK YOP . 



Statements 

1. The purposes of this project 
were clear to me at the start. 



Strongly' Un- Dis- Strong? y 

A^ree Agree decided agiee Disagree 



2. The objectives of this project 
are not realistic within the time 
allowed. . . . S 

3. The participants (students) are 
accepting the purposes of this 
project 

A. The student selection criteria 

have generally been appropriate. . . 

5. We the staff, are working together 
as a group ....... 

6. My 'time is being well spent, 
compared to other assignments I 
have had in this school district . . 

7. The program met my expectations^ 

' and I feel It Is providing something 
••unique*' to these youngsters .... 

4 

8. Ov»rall, I would say this project 
has betn successfully implemented 
during th^ first ypar , 
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9. Please cite; one or two specific . reasons for your response in question (8); 
especially If you ranked at 5 or 1. 



' 10. Please describe specifically any changes you would see as important to 
Improve the functioning of the project; either in terras of services to 
students or -general admlnlstratiye items. 



T 
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CLAYBUm CITY SCHOOLS 
EXEMPLARY WORK PROJECT 

EMPLOYER EVALUATION 1975/76 



Dace 



Your JUtmn 



Your Business 



5; 



I: 
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Name of Interviewer 
1. Names of EWF Student (s) in your employ at this time: 

b) ^ I 

' ' d) — 1 



^. (Use an additional form if more than above •) 

• ■ . ♦ ■ 

" 2; Please circle letters In (1) which correspond to those students covered in this 
report. " ' - 

3. How many E!VP students have, you had in. total since you became Involved with our 
program? . „ ,^ 

A', If you have had a turnover ef students during the past months, please cite the 
fia^oris for studerit termination, 1 

a) ■ ■ ■ - 



h) ■ 



d) ^ 



How vere you contacted about employing our students? 

\, fey phone 3; Hbkicft from supervisors 5. Counselor Referrals 

2; l^*rsonai contact A*- if^wspaper 6. Other 

fehis time what ts your overall fcacticin regatdihg. the present ith perfornance of 
Bbr Student(s) (circle proper number rh Scale) • 



l^dSikive 



4 



Negative 



|ft Stmii scrti^i piedsi teacfe td the job ptrfdrmdhce at the time the studentC?) 



•I 

^SSiliVe 



I 

n 



Negative 



8. Do you f«el Inclined to want to continue cooperation In this program beyond this 
present school year (assuming subsidized salary Is still possible)? » 

< ■' • > 

Yes . No Undecided 

9. Do you feel there Is a pojislbillty you might retain your "employee" beyond the 
termination of the subsidized funding? 

Ye« No • Undecided Not applicable 



lOe If the waver to question 9 Is No , please give the reason, 
. Unsatisfactory EWP student performance 
Students under age \ ' 

^.Job slots will not be available 

Insufficient funds of your own 

Other reasons (please list) 



11. Has your relationship with the Exemplary Work Project staff been satisfactory? . 
Yes No Undecided 

12. What problems have you detected In the project that need attention? 
Student pay checks late 

Confusion with required paper work 

Lack of communication with EWP staff 

Student job attitudes 

"^tudent pre-job orientation 

Student reliability , 
Other items (please list) 



13. Coments (positive or negative) 



t ■ » 
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Location of Interview 



1975 - 1976 

CLAYBURNE CITY SCHOOLS 
EXEMPIJVRY WORK PROJECT 

Parental Survey 



Date 



Name of Interviewer 
1. Parentis Name . 



2. Your Student's Name 



3- 



Highest Grade Your Student^ ha s Comp leted_ 
His/Her Previous School's Name 



Has he/she ever been expelled or exejapted? Yes 



No 



5* When wds the last time he/she attended "regular" school?_ 
5/A Wh6re does he/she attend school now 



6* How long has he/she been in the exemplary work project?^ 
?• l^ere is the present work station located? 



^. Does your student appear to be satisfied with the program up to this time? 
Yes No Undecided 

To the parent: Please check the appropriate box in the eight questions below - 
then please answer the last two questions in your own words. 



9. When my child was last in regular 
school he/she was enthusiastic about 
it at home. 

10. So far this year I have noticed a 
change for the better In the attitude 
of niy child. 

11. "Overall, in the exeinplary work project 

the performance of my child appears 
satisfactory. ^ 

12. I feel I know quite a bit about the 
goals of this project. 



13. 



I feel I' have been fully informed by 
the project straff relative to my child's 
progress. 



STRONGLY 
AGREE 


AGREE 


DISAGREE 


WJDECIDED OR 
DON'T KNOW 






























a 
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14* I feel there Is a good degree of 

cooperation and cornmunlcatlon between 
the parents t students, and staff. 

15; So far, I have been able to establish 
a positive opinion of this project 
regarding Its difference from "regular" 
schools* 

16. It appears as If ^the concept will 
meet (or has met) the particular needs 
of my child* 

17. In what particular ways do you think 
this projecf'^as affected your 



•STRONGLY 
AGREE 


AGREE 


DISAGREE 


UNDECIDED OR 
DON'T KNO!/ 










• 






t 

/ 




\ 
\ 
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18. Do you have any suggestions for change or Improvement In our project? 
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CASE STUDY NO. 7: NORTHWEST CAREER HIGH SCHOOL 



About 1970 the concept of career education was conceived *(or as 
some people say^ was revived from a beginning at a much earlier point In 
time). In any event the movement swept the country very rapidly and, 
while the concept was there, the basle definition arid operational format?^ 
were left up to Individual school districts or other educational agencies* 
Many different Ideas were, and still are, tried throughout the United 
States. Two events occurred In the Clayburne Unified School District 
that put them Into a position to develop a uniquely Innovative concept In 
career education with strong vocational avertones. 

The first of Jthese events was the realization by the then deputy 
superintendent and assistant superlntendejit that a career education unit 
should be formed for dlstrlct^wlde development of the total program. This 
unit employed a director hired from outside the district and subsumed the 
existing vocational education activities under Its purview. r 

The second event leading to the development of Northwest Career 
High School was the availability of a small school site. /The site had 
previously been used for another purpose and had dwindled In student 
enrollment so that It was no longer economically or operationally wise to 
operate It. Thus the availability of a site plus the evolving career concepts 
and top management support led to a newly structured instructional arena. 

In its effort to provide. an alternative to the ^traditional senior 
high school experience, Clayburne *s Northwest Career High School was 
opened in September, 1973, as an experimental project funded in large part 
by ESEA Title III funds. The school's career-oriented outreach program 
was directed at students who did not plan to continue directly . on to 
college; with instruction organized around occupational training, on-the- 
job training, and placement. A great deal of importance was also attached 
to the development of "career concepts" through a combinatioi^of classroom 
Instruction, intergroup relations, and personal counseling. At the same 
time, students complete the basic acacferaic subjects required for a standard 
high school diploma. This allows them' the option of entering a two-year 
community college, or even some four-year colleges, should their post high 
school career plans change. 

Northwest School is located in northwest Clayburne, the city's 
predominantly Black and Mexican American area^ Prior to 1973, it operated 
as a continuation high school with a student body that was drawn almost 
^exclusively from the local minority population. The school's reputation 
was one of violent students and an embattled teaching staff. "Little, 
if any, learning was taking place," according to one district report, and 
"many who were close to the situation felt that a drastic change should 
be made in the school or it should be closed." The assistant superinten- 
dent for secondary schools, William Lander, summed up the distric*:*s 
frustratipn with Northwest by declaring that the continuation program 
*Vas failing botl\ student/s and parents." 
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The Title III funds » for a three year period » permitted the backup 
for a complete changeover to the career concepts It allowed the employment 
of four additional staff members one of whom was the academic coordinator, 
one of whom was the occupational training coordinator, one of whom was the 
on*Job-c raining coordinator, and finally one who specialized in career 
counsellQg. The Title III award mandated evaluation each year and a 
considerable degree of latitude in creative evaluation was allowed. This 
perhaps was because standardized testing was not considered the most 
important method of evaluation by the district. Instead a number of non** 
test indicators were emphasized, some 6f which are described here. As an 
aside, for those reeders who i^ay be involved with this type of project 
during 'their evaluation careers, the inportance of standardized test 
results should not be Ignored. At the end of the third year the. district * 
applied for an Incentive (dlsseminatlor.) grant and It was denied. In 
retrospect. It was undoubtedly the low profile accorded to testing that 
encouraged the state to make a negative; decision. Historically, they have 
placed high emphasis on test scores and this posture continues. The 
evaluator seeking to Impress funding officials should place high priority 
to test results as well as to their presentation In graphic display. The 
state has required forms that must be used, however these may be supplemented 
by the evaluator* s own creations to highlight positive results. An example 
of how these kinds of data may be displayed was shown In case study number A. 

Standardized Achievement Tests such as the CTBS (Comprehensive Test of 
Basic Skills) are generally not used In purely vocational program evaluation. 
The criterion-referenced test; often developed especially for the project, 
is usually more appropriate. These tests would be designed to measure the 
student on specific levels of proficiency as demanded by the occupation In 
question. To repeat, the evaluator who Is operating with federal/state funds 
may %d.sh to be more emphatic In his use of nationally normed Instruments. 
Adequate descriptions and methods of use may be found In any recognized 
general purpose evaluation text. 

Technique 9: Case Study (2) 

The original needs assessment for the Northwest proposal was augmented 
by Information supplied by Dr. Lamar Gerry, who was one of the dlstrlct*s 
full-time ^valuators and a specialist In the design and administration of 
vocational education programs. Dr. Gerry surveyed a five-school sample of 
over 200 secondary students, as well as reviewing pertinent school records, 
to profile the expected population of Northwest In Its first year as a 
career center. Because the approach was to be significantly different 
from vocational programs elsewhere In Cla'yburne, and. Indeed In Its state, 
the closest thing to a prototype found by the proposal writers was the 
example of Skyline School In Dallas, Texas. For all those involved then. 
Northwest was to be an Introductory experience and very much ar experiment. 

' Profile of. Who Is the Evaluator? 

Dr. Lamar Gerry had been with the Clayturne district for some ten years 
as a teacher, vocational education specialist « and now program evaluator. 
Gerry's strength was In vocational or career education rather than evaluation 
per sc ; in fact. It was his expertise In the former that led to his appointment 
to the district's evaluation unit, primarily In charge of the various 
occupation-oriented programs operating In the city's secqnd^iry schools. 
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This background had strongly Influenced Dr. Gerry's approach to 
evaluation. Most apparent is his definition of the role he should play 
in a program, which closely follows state and district guidelines for tl\e 
position, Gerry saw himself as a neutral, outside $ource of information 
on the program objectives — first in the shaping 6f the objedctivef, and 
secondly on the progress the program is making towards their accomplish- 
meat. The means he chose to measure those program objectives also reflect 
Dr. Gerry's training. His conception of career ec^ucation, and his prior 
experience with it, predisposed him to a reliance on various qualitative, 
site level, and non-standardized measures of effectiveness; he felt that 
the qualities that Northwest was trying to develop simply could not be 
capture^ by scores on academic achievement tests. Reinforcing this 
disposition was Dr. Gerry's lack of a strong background in/'quantitative 
data ana^sis for evaluation. fHe stated th^t the s^lls^e«idid possess^ in 
this area had been developed without professional training in evaluation 
techniques. At the stame time, Gerry pines the CTBS testing schedule at 
the top of his list of priorities because it is required by law, ai^d 
thus the program evaluator's obligation. 

Dr. Gerry's background was also felt outside the evaluation. His 
expertise in vocational education gave him a unique role in the program 
design and implementation phases; and his personal commitment to the 
career education concept helped establish the strong communication links * 
he enjoyed with the principal (s) and program staff. 

This combination of skills and experience Iserved Dr. Gerry particu- 
larly well in the evaluation. He was able to $trike a balance between the 
professional limitations on his role .and the great persona]^ interest he had 
in the program's success. In concentrating on; providing the klRid of 
information which was immediately useful for program improvement.. Dr. Gerry 
contributed to the solidification at Northwest/ and its later adoption by 
the Claybume district. Had his emphasis been placed on what he. cited as 
the top priority — CTBS scores — the program might have been received 
differently in the State capital. But it woujld not have been the same 
program. 

On-going communication . i 

Dr. Gerry and both Nort^est's principals (^ere was a change midway 
in the second year) — especially Mrs. Warner (the*, second principal) — 
quickly instituted a series of formal and in|formal procedures to assure 
that the evaluation's Information got to thei site decision-maker as soon 
as possible. Gerry kept the principals acquainted with the instruments 
he was^sing; he met with Mr. Alcott (the first principal) and Warner 
whenever a survey or test had been completed; his observation visits to 
the program usually begyn or ended with a discussion in the principal's 
office; and, in the case of Mrs. Warner, the mid-year and year-end reports 
were reviewed by the evaluator an^ principal ttigether before being finalized. 
For the kinds of decision being made there, this rapid, constant provision 
of evaluative information was vital. That the communication channels were 
systematized so early in the life of the program can probably be aVtrlbuted 
to a congruent assessment of information needs by the evaluator and the pro« 
gram )director(s) , and (in no small part) to the personal compatibility of 
Dr. Gerry with Northwest's two principals. 
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The cpnununl cation between the evaluator and state or district author* 
Itles was less frequent and more formalized. Dr. Gerry and Mr. Isley* the 
State Coordinator « met several times a year to discuss the project « but 
written communication — memos from Mr. Isley* the interim and final reports 
from Dr. Cerry — were more common. The same pattern held with the flow of 
information to the district officer. Gerry met Informally with ^personnel in 
the Secondary Schools and the Vocational Education divisions « but their main 
link was the written reports at mid-year and yearns end. For the decisions 
about the project that were made by these two levels of school governance, 
on-going communication as such seemed to be much less Important than the 
different kinds of Information that each agency sought. 

V 

The majority of the preceedlng description is excer|^ted from a 
larger case study done on the project. Technically the proper way to do 
a case study is to get viewpoints from at least two vantage points and then 
consolidate and synthesize the results into a composite whole. In this 
case the steps were as follows: 

• Identical questionnaires given to the evaluator and the two 
school principals.' None of them saw how the others answered. 

• After an analysis of the questionnaires « the third party researcher 
interviewed all respondents. 

. A series of further in-depth interviews were conducted with the 
evaluator. 

• A final document (case study) was prepared and submitted to 
evaluator and principals for approval and annotation. 

. A final closing interview was held* with all parties, present « in 
in order to reach consensus on the validity of the case study. 

Technique 10: Process Evaluation 

Process evaluation is that technique whereby information is fed to 
the project decision makers during the course of operation so that 
directional changes may be made if necessary. It was Illustrated in the 
description of Northwest Career High School how this was accomplished by 
the evaluator. This technique is most effective when the personality of 
the evaluator lends Itself to ease of communication with the project 
director. The evaluator « above all« must not be considered as a threat; 
he should he viewed as a functioning member of the total instructional 
process. It is up to the evaluator to develop technical skills and human 
relations skills that will make him truly effective As a "process evaluator. 

Advantage of Process Evaluation: 

a. The principal advantage of pi^ocess evaluation Is to feed 
back to the project director suggestions for program 
improvement during the course of operation so as ^o 
improve instruction while the intervention technique is 
actually taking place. 
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DiBodvantage of Prooese Evaluation: 

The organizational structure or the personalities of the 
evaluator and/or project director may preclude meaningful 
rapport and utilization of *'in-progress'' results to help 
students. 

Evaluation Forms 



Some of the project-developed forms for use in eliciting te^onses 
from major project participants are on the following pages. These were 
used for primary participants; students and faculty » and for parents as 
secondary participants; one record keeping form is also displayed. 
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Interviewer 



Date 



STUDENT INTERVIEW 



Naffle 



Age 



Grade Level_ 
Clty_ 



Street Address 



^' ^^^Vh""" ^"/""I'T^^"'^ ^" )Quarter: (Please Indicate quarter 

(Include occupational. cluster to )Sen.ester or seniester by crossing 

which assigned, as well^as- any ^ crossing 
on- job training) . 



out- inappropriate tern.) 



i 

_ (period- 1) 
.(period 2) 
.(period 3) 
.(period A) 
.(period 5) 



.(cluster) 
(OJT) 



2. What plans have you made for a job after you finish school? 



3. Who has helped you the most, either here at school or outside of school 
In planning for a job or for planning an educational training, prograr.? * 



Re^ to the student ; "The following .information is needed to aid In estab- 
lishing and maintaining quality career education pro- 
grams with equal opportunity for all groups." 

A. Which of the following do yoo consider yourself to be? 



American Indian 
Black 

Mexican American 
Oriental 
Puerto Ricnn 



Spanish American 
White 



Other (specify) 



/ 
I 



5. What do you like best about your present school situation? 
Probe: Do you get along OK wic| your classmates'? 



6. Whftt do you dislike most about your present school situation? 
Ptobe: Do any of |he students give you' a hard time? 

7. Overall, how do you feel about school this year? 
Probe: Is it any better or any worse than last year? 



8. Have you made any close friends at this school? 
Probe: Who Is your best friend here? 



9. What do you wish was different about this school? 



10. (For transported students only) What do you think about riding the bus 
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CLAYBURNE CITY SCHOOLS 
NORTHWEST CAREER HIGH SCHOOL 



Faculty Evaluation Scale On Present Status 

Of Scliool 



DATE Did you teach at Northwest 

last year? Yes No 

(Please do not sign your; name) ^ (Answer is optional) 
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Directions ; Read each statement carefully and decide how you feel about it* You 
will d{{ree with some' statements and disagree with others.* You are 
« offered five possible answers to each statement. The' "undecided** 
answer should be circled only whea you have no opinion. Circle one 
number following each statement, Pleasp answer all statements , and 
record your feelings as of this date. 

Statements : Strongly Un- Dis- Strongly 

Agree . Agree decided agree Disagree 

1, The purposes of this school 
are clearer to me now than 

three montjhs ago 5 ^4 .3 2 1 

2, The objectives of this school 

are not realistic within the ' 

time allowed 5 A 3 2 1 

3, The faculty are accepting 

the purposes of this school 5 A 3 2 1 

» • 

A, I am learning m9re about 

career education as the year 

progresses .^ , , , 5 A 3 2 1. 

5, Ue are working together as a 

group 5 A 3 2 1 

6, I really feel a part of this 

group now 5 A 3 2 '1 

7, The program is meeting my '/ 

expectations , , , 5 A 3 2 1 

8, Relative to my teaching • . , . 
experiences last year, this 

is more rewarding 5 A 3 2 1 

9, Please list below any particularly good points or bad points of the schooi as 
you perceived them. 



10, Please list below any specific Ideas you have for school Improvement durlnp; the 
balance of the year, ^ 



CLAYBURNE CITY SCHOOLS 
NORTHWEST CAREER HIGH SCHOOL 



Parental Survey 



C;;rcle One: 
Pre Post 
Your Name 



Date 



Your Student's Name 



Hls/Her Previous School's Name 
DIRECTIONJ 
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Belov are listed eight statements regarding your knowledge of this school as well as 
your child's reaction toward It* Opposite each statement you should place a check 
mark to Indicate your feeling or your opinion of It. Note that the possible choices 
arc: STRONGLY AGREE, AGREE, DISAGREE, QR UNDECIDED. After you have completed the 
first eight questions, we ask that you respond to questions 9 and 10 in your own words. 
Your comments will help uS to Improve the school In line with the needs of your child. 
THANK YOU FOR YOUR HELP. 



1. 
2. 

3. 
A. 
5. 
6. 
7. 

8. 

9. 



When my child was In school last year , 
he/she was enthusiastic about It at home. 

So far this venr I have noticed a change 
for the- better in the "school attitude" 
of my child. 

So far this year I have not noticed 9ny 
change In my child^s attitude toward school. 

I feel I know quite a bit about the goals 
of the Clayburne Career High School. 

I foel I have been fully Informed by the 
school staff relative to my child's progress. 

I feel there is a good degree of cooperation 
and communication between the parents. 



So far, I have been able to establish an 
opinion of this scliool rcVjarding its' differ- 
ence from other schools* 

It appears as If thi.o school will meet (or 
■let) the partlculaj needs of my ^hlld. 

In , what particular ways do you think this 
school hati aff»;ctcd your child? 



STRONGLY 
AGREE 


AGREE 


DISAGREE 


UNDECIDED 








• 




« 




• 


























• 






















• 



O 10. 
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Do you hftvc nny oiin>',ostlona for cbnnp,o «»r Improvement In the 8cl>ool? 
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NORTHWEST CAREER HIGH SCHOOL a. 



COMPARISON OF MARK POINT AVERAGES^) 
STUDENT. DATA COLLECTION ROSTER 



Date 



Data Collected by_ 



4 

STUDm NAME 


PRESENT 
OCCUPATIONAL 
CLUSTER 


PRESENT, 
GRADE 


SEX 


MARK POINT 
AVFRARF 


SCIIOO 

IN Jir 


L ATTENDED 
NE 1974 


REMARKS 


6/74 


2/76 




(OTHER) 


e.g. 

Jones, John 


Construction 


12 


*M 


2.8 


3.1 


yes 




1 


Smith. Jane 


Urban A^rrlc. 


12 




2.4 


3.4 


•, 


Hoover 


1. 
















2. 




• 












3. 




• 












4. 














t 


5. 














• 


6. 


• 














, t 


7. 




4 












8. 


• 














it 

9. 
















10. 






• 










11. 
















% 

'■ 


12. 
















13. 
















n. 














• 


15. 

















mi»X have been enrolled prior to then. 



" but 



i 
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STUDY QUESTIONS PART II 

a) Develop a questionnaire on any eva}.uation subject, mall 
to 30 homogeneous people, and analyze the results « 

b) , Develop , a structured interview questionnaire. Use it to 

interview six people in a given category of vocational 
training (or delivery system), and write a two ]^&ge 
' descriptive report. 

c) Analyze results of recent research and write a two page 
paper arguing either for or against the improvement of 

' basic skills (reading and mathematics) through the use 
of vocational or career education*. 

d) Develop your own follow-up questionnaire, mail to 30 
ex-students and analyze results. 
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PART III 



EVALUATJON ISSUES 



1. Evaluation Design & Decision Mak^n8 

Evaluators should h^ve considerable Input Into the original design 
of a project which will ultimately require his/her services In gathering 
data and preparing Interim and final evaluation reports* Within a local 
school district supporting an autonomous and adequately staffed evaluation 
department, the degree of service will naturally vary with the individual 
evaluator Involved. The very worst that can happen (and all to often, 
does happen) is that the assigned evaluator is never consulted on the 
original project construction. Is brought Into the picture after the 
fact, and Is delegated responsibility for evaluation at some point 
after the project funding has occurred. 

The next best thing Is that some evaluator (no^ necessarily the one 
ultimately assigned to the project) is consulted during the early project 
writing stages, and Is at least requested to examine the objectives and 
be sure they are '^risp enough" for measureiient in the future* The 
evaluator, however, should caiitlon himself not to be too hasty in making 
Judgments on the efficacy or worth of the objectives; his job is. basically 
tb see If they are capable of quantifiable measurement. The best that can 
happen within time constraints and interests of the evaluator is that he be 
intimately Involved In all. aspects of the project application development* 
Projects directed to federal and state funding agencies are often written 
by relatively Inexperienced writers who have no real knowledge of the problems 
to be addressed by the application. Therefore, the evaluator, if he or 
she Is experienced, can be of immeasurable help during these early stages. 

A 

People can deal with problems more effectively If they can anticipate 
the nature of those problems. For the educatloail evaluator it is Imperative 
to point out that most educational evaluations vLll be carried out in a 
thoroughly practical milieu in vhich the results may constitute playing 
cards that people will be dealing with from potentially political decks. 

Evaluators tend over time to improve the quality of their data gathering 
and report preparation. Educational evaluators should feel fulfilled if. 
In their opinion, they have been able to enhance the quality of educational 
endeavors to which they dlrc^ct their attention. Many evaluators initially 
assume that the educational ♦orld is out there, feverishly awaiting the 
results of their studies. Too .often a surprise results, and It becomes 
apparent that evaluation, especially that which Is required by extramural 
funding agencies, is all too often a rote process, and the results count for 
very little In the ultimate decision-maldng procedures. 

The evaluator can have a significant effect on local process decision 
making it he has a personality that meshes with that of the project director 
and if his ongoing data-gathering efforts are used by the district for 
program improvement. At the state level, however. It is doubtful if his 
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efforts ©ay ever amount to much more than a preparation of the standard 
forms that are required by the various funding sources. These forms usually 
bear very heavily on theluse of pre-post standardised test results and all 
too oftten do not take cognizance of the non^test Indicators that can so often 
assist the director In establishing a feel for "the real pulse of the operation." 
States also very often make funding decisions using data from sources otlier 
than what Is provided by local program evaluators. ^ ^ 

2. Preparing a Needs Assessment 

• • 
Presumably evaluations are conducted on projects that are designed 
to cope with an educational need, or at least a need as perceived by 
administrators of an educational agency. Funding agencies usually require 
a fairly formal needs assessment to be performed during the Initial stages 
of preparing a proposal for outside monies. Even If a school district Is 
developing Its own Inhouse vocational program a good needs assessment Is 
desirable. The needs assessment can b^^ developed In a number of ways, but 
the essential elements are to state the existing condition and Indlc^^te how 
the project will take students to the desired condition. For exampl^: 



EXISTING 
CONDITION 
(what Is) 

Approximately 
10% of high 
school students 
In Bayvlew public 
schools are pot- 
ential dropouts 
and also have 
expressed Inter- 
est In an alter-* 
native type of 
school. 

The majority of 
these students 
are not enrolled 
in a vocational 
class. 

On the average 9 a 
random sample of 
these students 
have reading 
scores below the 
median. 



DESIRED 
CORDITION 



At least^30p of ^ 
the students 
identified as 
potential drop- 
out s« non voca- 
tional enrolleeSt 
and 9 low academic 
performers be 
enrolled in an 
alternative 
school to be 
known as the 
northwest career 
education high 
school. 



'DISCREPANCY 



Ho such alterna- 
tive school 
presently exists 
in this school 
district. 



ANALYSIS OF 
DISCREPANCY 



An unused site 
presently exists 
that would be 
adequate for such 
a school The 
district lacks 
adequate financial 
resources to staff 
such a school with 
the specialized 
personnel that are 
required. Addi- 
tional support funds 
would enable the 
district to imple- 
ment this new con- 
cept. 



Information to v 'idate the existing condition may bo gathered by the 
use of questionnaires sent to parents, staff members, employers, or students. 



CENTRAL CITY SCkOOLS 

SCHOOL IVPROVEMSIiT PROGRAM 
HOU ' JS HIGH SGHCOL 
WEEDS ASSESSKHNT PRCFILS 



Dtar Parontt 

V« need your help. Our school Is continually assessing needs In order to develop 
appropriate educational pro-^rans for childrent We want to know your views in regaid 
id their needs. We invite you to take part in the planning by completing this brief 
questionnaire. Please nstum it to the school as soon as possible. Thank you. 



My child is in grades t 



10. 



12 (Please check one). 



DIRECTIONS I Below are list^ 20 items, ten of them in academic areas and ten of 
then in non-academic areas. Please rank each one on a scale of 5 " 1 (high to low) 
depending on how you perceive its importance to the instruction of children at our 
school. Please circle opposite each item the dumber that best describes your feelings. 



ACADEJCTC C 




high/low 




NON ACADEMIC 




high/low 




1. 


Reading 


5 


4 


3 


2 


1 


1. Development of Staff 


5 


4 


3 


2 


1 


2. 


Oral Expression 


5 


h 


3 


2 


1 


• 2« Student Self-Concept 


5 


4 


3 


2 


1 


3. 


Written Expression 


5 




3 


2 


'1 


3« Physical Education 


5 


4 


3 


2 


1 


4. 


Mathematics 


5 




3 


2 


1 


4, Health Education 


5 


4 


3 


2 


1 


5. 


Science 


5 


4 


3 


2 


1 


5* Counseling 


5 


4 


3 


2 


1 


6. 


Vocational Education 


5 


4 


3 


2 


1 


6.. Parent Education 


5 


4 


3 


2 


1 


7. 


Social Studies 


5 




3 


2 


1 


7. Teacher Conferences 


5 


4 


3 


2 


1 


«. 


Bilingual Studies 


5 


4 


3 


2 


1 


8« Career Education 


5 


4 


3 


2 


1 


9. 


Art and Music 


5 


4 


3 


2 


1 


9; Consumer Economics 


5 


4 


3 


2 


1 


10, 


Othet Enrichment 


5 


4 


3 


2 


1 


10. Students with Special 


5 


\ 


3 


2 


1 



Needs 

Are there any other areas in which you feel our school needs to place special attention? 



OTHER COKMSNTSi 
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PI£ASE RSTURil PROOTLY 
89 Thank you. 



Thlt Is one way of gathering information, but not the only one. The 
principles of questionnaire construction discussed in the section on 
follow-up should be adhered to with special attention paid to brevity and 
ease of tabulation of data received. One example o^ a short parental nc '3 
assessment, from which" excellent results have been achieved, is shown. 

J 

Parental, staff, employer, and student input is very important. 
However^ this should be, supplemented by information gathered by unobtrusive 
means. For example school records of attendance, grade point averages, test 
scores, and referrals* to counselors should be used whenever possible. In 
larger school districts much of this information is contained in the computer 
data bank ^d may be obtained without resorting to undue consumption of 
peoples* tibe and attendant possibilities for biased responses. 

3. Is The Measurement Instrument Appropriate? 

Throughout the manual we have referred to various measuring instru- 
ments, whether they be standardized tests, criterion tests, structured * 
Interview^ forms, survey forms, or other self-developed devices. Each one 
hds its place and is usually tailored for a specific vocational program. 
Ot it may be modified from some other instrument that has been located 
elsewhere. If modification is the technique, be sure to watch the copy- 
right laws and assure yourself that you are making no violations. Often 
times a letter to the publisher or the developer of another test will 
provide the necessary clearance to "borrow" a few Items. 

THe approprlatenesii of instruments for use in vocational evaluations 
is especially difficult to determine. Obviously they must be related to 
the subject matter beinj; taught (or the attltudinal development being 
sought). The Instruments should have other characteristics too, such as 
ease of administration, scoring, and interpretation. Teachers who use 
any evaluation instruments, whether commercial or self -developed, shduld 
have an opportunity to rate them after having had an opportunity to try 
them In the classroom. A useful scale Is shown. This scale. In Itself, 
was a project-developed instrument, which borrowed on experiences of both 
teachers and evaluators. 

4. Abstracting the Report for Reader Utility 

Evaluation reports often tend to become quite long which may discourage 
busy administrators from reading them In toto. It therefore is a good 
idea to condense the essential elements of the report on one or two pages 
at the very front of the document. This permits the decision makers or 
other interested persons to receive immediate Information on t\\e results 
of the evaluation ascwell as the main thrusts of the project whfle it was 
in progress. Unless an extra curricular funding agency requires a certain 
format, the evaluator may design his own abstract page(s). In any case, 
no matter what the outside source requires, it is usually wise for the LEA 
to design its own document for use within its family of administrators and 
decision makers. Two possible abstract designs are shown; one confined to a 
•ingle page and one somewhat longer. 
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CENTRAL CITY SCHOOLS 

TEACHER RATING SCALE - STUDENT TERTS 

NAME OF TEST: ; DATE RATED: ^ 

TYPE OF STUDENTS FOR WIIICH IT IS APPROPRIATE? 

GRADE LEVEL OF STUDENT FOR WHICH IT IS APPROPRIATE? " 

ADMINISTF.UKD: INDIVIDUAL CROUP EITHER ' 

TIME REQUIRED FOR ADMINISTRATION? 

NAhE OF TEACHER COMPLETING THIS FORIl: 





DIMENSIONS HIGH 

•• •' " 5 




rati:; 

J 


C 


LOW 

1 


RELEVANT QUESTION 


1. 


Validity 

f 


5 . 
% 

5 


A 


3 

« 


2 


1 


Does the test appear to measure 
your specific educational objec- 
tives? 

• 


« 

2. 


> Validity 


A 


3 


2 , 


1 


Does the ^st appear to predict 
the student *s performance on 
related tasks? > 


3. 


Undcrstandlblllty 


5 


A 


• 3 


2 


1 


Is the coTiprehenslon level 
correct for the a^c and educa- 
tional level tp which the test 
Is dlrcctcd?'^ 


A. 


Content 
Usefulness 


5 


A 


3 


2 


1 


Is the test content closely 
related to your curriculun? 


5- 


Ease of 

Administration 


5 


A 




2 


1 


Is the tasteeaslly and conve- 
niently adSnlstered? 


6. 


Scoring 


5 


A 


3 


2 


1 


Can the test be easily and 
reliably scored by hand? 


7. 


Interpretation 


5 


A 


3 


2 


1 


Can the scoreis.be easily inter- 
preted and decisions be cade 
from the results? 


8. 


Reliability 


5 


A 


■ 3 


2 


1 


Does the test measure consis- 
tently from one tine to another? 
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Conments (particularly any major strengths or weaknesses): 



CEIITRAL UNXnSD SCHOOL DISTRICT 
EVALUATION REPORT ABSTRACT* 



TITLS or PROJECT t DATEi March 12, 1978 , 

Busineu Careers Training FUNDING t E^EA «nd District in ^ " . 

for Disadvantaged Students kind ^ ^ ' 

S3D ReT)ort Ntm'bert 188 "~ ' • i : ° • " 

DESCRIPTION 05 PROJECT i ' Business Careers training was designed -^o Impart *ntry-level 
^klUs to 300 eleventli grade students from the Linda Vista area of Central City. It was 
operated In two high schools during the 1977-78 school year and utilized the individualized 
learning packet appifoach with a heavy emphasis on peer tutoring. The 183 boys, and 1,17 girls 
who entered the progAn in September 1977 were especially selected through administration of : 
the Career Planning Inventozy by each schools' cbunsellng staff. 

■ •* ■ . i » 

MAJOR PPXRAX OBJECTIVES t As a result of successfully participating in the business program-.' 
each studerit will be provided with an entry-level skill which will enable him to type 60 words ' 
per minute and file ,120 letters per hour* . Seventy five percent of those stiidents w^o enteir 
the program will s]|pcfsssfully acquire this skill as verified by the Occupational Skills 
Inventory, administered dus^ng the fotirth ^uarte;r of the year. Fifty percent of those students 
requesting help will be placed in summe): Jobs related to their classroom training'. 

EVALUATION DESIGN t Students were pre-post tested with the OccupatiorJil Skills Inventory over^ 
the period October 1976 to June 1977* Oplnionnaires were adm^^nlstered to' students, teachers, 
8 lie' administrators, and local employers. Finally, an assessment aind validation of*th^ 
curriculum was performed! by the industry*based Central City Businessmens'- Occupational 
Advisory Committee. ^ ^ 

EVALUATION FINblNGS t It was found that 82^ of those students who started the program completed 
it by the end of the school year. Of those who completed the work there was' an entry-level 
skill certification <sf S^%, The most easily acquired skill appeared to be the filing of'^ 
letters, while the ability to type 60 words per mir.ute appeared to be correlated with the level 
of disadvantagenent and was attained most easily b^* sti^ents who scored At the 76th percentile 
. 3r higher in reading tests. < ' • . 

— : ^ : A ^ _ . : , 

RECOyKENDATTONS i On the basis of the evaluation findings It is recpmmended that these actions 
should be takens * . 

1. Involve higher student selectivity in the initial counseling. * * 

2. Considei* the possibility of using another standardized test for pre-post evaluation. 

3« Refine the techniques of identifying disadvantaged students. o 
.± U . : ■—, _ 

4 

This pa/^e Is to be inserted iianediately behind the Table of Contents in all Evaluation 
Services Department reports .'^ 

. • . ■■ . . 
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SUMMARY OF FINDINGS AND RECOMMENDATIONS 



^Conclusions 
Reconanendatlons 
Project Objectives 
Profile of the Project Participants 
Profile of the Project Lpcale 

{ 



EXEMPLARY WORK PROGRAM 



1975-76 



CONCLUSIONS 



The following conclusions have been reached^^as a result of the Exemplary Work 
Program review and analysis: 

.1. The data suggest that project participants, who had prior records 
with the Probation Department t had a recidivism r/site about 10% 
lower than what was anticipated; although the time period has 
"qualifications". 

.2. The data show that there were no arrest? among project participants 

who Mete valid "non-prlors." Calculated probabilities Indicated 12% 

could have reasonably been expected to be arrested. 

f 

3. All Indications are that paid employment Is helpful In Juvenile 
crime reduction and that a cause/effect relationship does exist. 

4. Enthusiasm for school, while still marginal, was enhanced in many 
cases by project participation. 

5. Records Indicate that about two thirds of th^ project .participants 
had previously had contact with the crln\lnal justice system, ' therefore 
the student selection was usually appropriate. 

6. Th^ student-participants benefit heavily from Intensive career and 
" Ally life" counseling. The correct counseling technique Is a 
key to program success, but further research on techniques is 
heeded. 



7. Strong community support was exhibited and could be employed to great 
advantage if a similar project is implemented in the future. ' 

8. The length of program operation was clearly too short to attain the , 
depth of desirable outcomes that had been anticipated. 

9. The total cost of the program was^^transla^d to approximately 
$1,730 per student participant. 

T 

10. A sound evaluation methodology was developed and could be transportable 
to future projects of a similar nature. 

11. There were gaps in comrcunicatlQn ^rom the state/federal level, as 
w)sll as Internal staffing problems that acted to lessen program 
Success. ; 

12. There appears to be a duplication and overlapping of ef f6rts in 
federally funded projects of this type in the same geographical area. 
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RECOMMENDATIONS 

On the basis of the conclusions reached In the Exemplary Work Program review 
and analysis, it Is recommended that these actions should be taken: 

1. Dialogue should be .initiated with apprd))rlate funding sources to . 
provide backing for a similar program to be operated for a minimum 
of three y^ars, start4.r43 In 1977. 

2. The evaluation methodology developed during 1975*76 be re Ined and 
updated for use in any subsequent funding years. 

• ' >^ ' 

3. The lines of communication and decision making points at state/federal 

levels be clearly established before starting another program. 

f 

ft 

4. Future program administration at the local level must be closely 
accountable for use of resources /and 'delivery of high quality career * 
counseling and job training. 

5'. The educational segment of such a program should be emphasized more 
heavily and student progress monitored more closely. 

Care should be taken to avoid dupllbatlons and overlaps with agencies 
performing similar functions, or at Xeast to work closely with, and 
communicate \ri.th, those agencies that appe;^r to offer related services. 

7. Community advisory committees be used heavily in any future program in 
order to avoid duplications, provide liaison, and help create' suitable 
Job slots — especially in the private sector. 

8^ In future programs the record^ keeping system, covering all facets of 
the operation, must be accurate and systematically available, to the 
evaluators in order to properly measuriB progress towards predetermined 
objectives. 

PROJECT OBJECTIVES ' 

The primary objective of the program was to take 600 teenage juveniles, residing 
In high crime rate areas of Bayview, and put them to work on paid jobs. 

A secon/ary objective was to provide the counseling and other support services 
necessary for fhe conduct of the program. \ 

A third objective was to attempt to prove a cause/effect relationship between 
youth unemployment and juvenile crime. 

• 

Specifically these objectives were translated into three major activities on 
which the evaluators were to collect data: 

1. Reduce by 12.52 the anticipated A0% recidivism rate of project 
participants with known prior Probation Department records. 



2« Reduce by 12 .52 the anticipated 12Z annual arrest rate of teenage 
juveniles, living In program service areas, who are known 
"non-prlors." 

3« Document the processes associated with program operation* 



The student participants consisted of approximately 550 Individuals during 
fullscale program operation* 

a 

1. The ethnic brealcdown was ^proximately 36% white, 36% black, 

13% Spanish surname, and B% other minority or unable to Identify. 

• 2. The age breakdown was approximately 12% of 15 year olds, 27% of 16 
year olds, 40%, of 17 year olds, and 21% of 18 year olds. 

3. The sex breakdown was 61% males and 39% females. 

•4 

The staff participants numbered between 27 and 30 people during the project 
operation.. The typical staffing pattern was*. 



Incentives for evialuatlon should Ideally come directly from the agency 
operating the vocational program. These incentives should revolve around the 
process aspects which will help the project staff improve the program as it 
unfolds, as well as summary data which will tell how well the program did 
(|>ased on its original objectives), and will provide a "base for continuance 
or discontinuance. This posture assumes that the local educational agency • 
considers evaluation of its vocational programs as an Integral part of the 
Instructional process. In order to assume this stance, the LEA must be 
trilling to commit a share of its own financial and personnel resources to 
this ongoing task. In reality, whether due to scarce financial baclcing or 
general lack of perception of value, this "non-motivated" evaluation is seldom 
accomplished. In the era in which we llve» it is probably understandable that 
this situation should be so. Teachers and the general instructional program 
compete for funds, and it is difficult to. ascribe the Immediate value of 
program evaluation. 



PROniE OF THE PROJECT PARTICIPANTS 



Project Director 
Administrative Aide 
Career Counselors 
Head Instructorc^ 
Hourly Instructors 
Teacher Aides 
Community Aides 
Secretary ^ 
Payroll Clerk 
Time Card Clerk 
Clerical Pool 
Clerk - Bilingual 



-1 
-1 
-4 
-1 
-3 
-2 
-2 
-1 
-1 
-2 
-8 
-1 



(ardent s) 
(students) 



Incentives for Evaluation 
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Vhat, then, are the major factors that dictate vocational education 
evaluation today at the local school district level? A desire to measure 
results and provide program improvement may be one of them, but It is not , 
paramount. The primary impetus is from the direction of the categorical 
funding source. The ESEA, Title IV-C requirements, as an example, require 
mid-*year and year-end evaluations along fairly structured lines on specific 
totms provided by the agency. As mentioned before, these forms rely heavily 
on the use of standardized test results, vmile the report format is very 
mtuch structured, the evaluator is not precluded from creating his own 
^'addendum" which may offer the opportunity for a bit more insightful 
description of program results. 

Another source of evaluation requirement, other than the state or other 
categorical funding agency, could be court orders involving desegregation, 
federal legislation, and other such situation^* The central issue is: does 
the local agency conduct evaluatior^s of ^i,ts own programs simply for the sake 
of making them better? Or, does the agency merely conduct e\^luations as a 
compliance for outside funding with local program improvement as; a possible 
**spln-off"? Too often the latter situation persists. 

6. Politics of Evaluation 

^ Three critical decision points exist in most vocational education 
evaluation. Since the state is the distribution agent for the federal 
funds (either VEA or Title IV-C), there is clearly a focal point here 
which may be divided between formative and summative issues. The state 
has certain rep6rts it requires, and it also conducts one on-site visits- 
titm each year during the finanQ^^dl support of a project. The state 
contact person and the evaluator may agree on the points that are needed 
for program improvement « except it inust be kept in mind that the state 
will always attach great importance to the Comprehensive Tests of Basic 
Skills (or other standardized measures). During the first two and a 
half years of project funding, the decision-iQaking process is most likely 
a dual one between the evaluator and the state representative. At the end 
of the third year, on the other hand, the state decision-making role shifts 
completely to a review committee. The review committee makes but orm 
decision: to extend state support for, project dissemination to oth^r 
schools or to terminate the funds er.lrely. it is Bt this point that the 
politics of* evaluation come into play. 

The next hierarchical level for political decision making is within 
the district structure itself. If the district employs^ a competent and 
experienced vocational education evaluator, his expertise can virtually 
preclude the intromission of other district officials in the formulation 
of recommendations. The competent evaluator should' be given full aut^hority 
and responsibility for program^recommendations. The wise evaluator, 
ho%pever, will always realize his poisition is just that — one of an evalua- 
tor* While he may have fall authority for making recommendations, it is 
ultimately the project director (who may be a school principal) who roust 
make the final decisions and ipplercentations. Like the>«tat;e process, 
however, a final decision to keep, eliminate, or modify a program In the 
district is made at a high level — the secondary schools division-^and in 
the case of outside funded programs, only at the end of three yeara. 
But, unlike the state, the district also takes into account the numerous 
program facets that are not reflected in test scpres alo|iet^f .ft^t^-stud^nt 
attendance and attitude, curriculum development, Itnvac^ ^the p^nanMnity^ 
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and* 80 on. The evaluators* reports can provide the Information n^essary for 
a study and continuous documentation of program progress to support a 
district decision to continue it after outside funding ceases. It^ is doubtful 
if many programs will survive after the three-year outside funding! if reliance 
is placed only on the structured state reporting forms. It is at :his point 
that the evaluators* creative skills, can bear fruit. He can creatf a 
descriptive report of his own design that will supplement the stat^ reports 
and will assist the district in its decision for the future. Not only a 
report, but the personality of the evaluator can be of paramount importance 
In the local politics of program operation. His technical evaluation skills 
are Important to be sure, but probably far more important at the district 
level is the rapport he/she has been able .to generate with the staff. This 
means the teaching staff with whom he deals, the project director, and members 
of the secondary schools staff at the decision-making level. Results are not 
guaranteed by a pleasing personality, of course, for the final decisions 
could ultimately be based on many other factors. The evaluator should play 
a role that closely follows state and* district guidelines for the .position^ 
and he shoulc' be a neutral, outside source of information on program objectives 
first, in the shaping of the objectives and secondly, in measuring progress 
towards them. His personality and stcff realtionships throughout this process 
can have important implications in. the politics at the local level. ^ 

The third decision-making point — after the state and the districtr 
level staff — is the school principal or project director. For internal 
purposes this is the one terminus for process decision making. The 
personal charfTcteristics of both the principal and the evaluator come into 
play heavily at this point. If the evaluator had demonstrated competence 
and» at the same time, is able to project a low key but cooperative profile, 
he probably caii emerge as a key person in program operation and changes in 
emphasis. In the hierarchy of the politics of evblfuation, the program 
evaluators* influence can probably be most deeply felt in his day-to-day 
relationships with the project operating staff. The evaluator would be 
well advised to content himself with successes at this end. 

7. Finding Your X)wn Role as an Evaluator ^ 

The conttnents which follow presuppose that the evaluator iA an 
employee of a public organization, such as a local schdol district. The 
eval^iator employed as a private consultant will need to find his role in 
a completely different manner. . \ 

Typically a vocational program, especially one with "outside funding," 
will have a project director. If the project" odcurs within a single 
school, or actually consists of the entire school Itself, the project 
director may often be the site principal. Projects that are broader in 
icope, cutting through several schools, usually have an administrator from 
the central district office designated as the director. In either case 
one or more Inherent problems of evaluator /director relations is almost 
sure to arise over a period of time. As has* been pointed out several times 
in this manual, the personality of ^the evaluator and the degree of his 
rapport is certain to come into play in* the resolution of conflicts or 
misunderstandings. Some of the situations that may occur for the evalua- 
tor are; 

♦ 
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Gaining Confidence and Credibility 

* d 
A number of questions fall Into this general rubric • Initially the 
activities and the motives of the evaluator may be very suspect; albeit 
he may well be ostensibly gteeted with open arms and guarded enthusiasm 
when he arrives on (the scene to discuss an evaluation plan with the director . 
Evaluation Intrinsically seems to have a threatening effect on project 
personnel, although the nature of the threat may vary between two types of ^ 
. employees;^ project directors (principals) and teachers. 

The principal has noted a budget line' Item In his project proposal 
or plan that Indicates some dollars assigned for evaluation, but unless 
he or she has had previous experience with special projects there Is probably 
little cognizance of what In entailed. On the other hand, this can be an 
advantage to the evaluator. If the director is inexperienced, he may be 
likely to place complete confidence in the evaluator and give him carte- 
blanche to design the overall scheme. In this situation the experienced 
and skillful evaluator is able to transcend his job description and provide 
significant process-input to help Improve programs. 

The good evaluator can obtain the confidence and respect of most 
project directors although it may take time. It is advisable to move 
slowly at first until such time as the director has an opportunity to 
recognize the skills possessed by the evaluator and the services he can 
perform. The evaluator, to gain complete confidence, roust be in possession 
of the technical skills necessary to do the job. If he does not know an 
answer to a question, he should be quick to say so, but he should be 
equally quick in saying "I will get you the answer to that question 
tomorrow." Project directors often need information in a hurry in order 
to make process decisions. 

Gaining the confidence of project teachers is probably more diffi- 
cult than for the director. Teachers see any evaluation as a threat and 
in the majority of cases they do not understand the need for it. They 
rarely are aware-of the. outside funding that makes evaluation a requirement. 
Also the amount of record keeping that may be required of them on special 
projects may not be very popular. * 

' Teachers are usually enthusiastic about the money they receive from 
a project for new equipment, classroom teacher aides,, or other refinements. 
They are often (but not always) less enthusiastic about the need for account- 
ability. Both from this point of view a«d the point of view of classroom 
instruction being disrupted, it is probably best that the evalua,tor 
collect as much data as possible through unobtrusive means. In this 
regard the computer with, its large student data bank, can be of great 
assistance in districts large enough to have this capability. 

Dollars Allocated for Evaluation 

' Normally a project will have somewhere between 5% and. 8% of the^ 

budget allocated for evaluation. The first several months of project evalua- 
tion rarely sees a questioninp; of this budpct item on the part of t;he 
director, especially if the evaluator is competent and ijppears at the 
project site freqISently enough to maintain what is considered good visibility. 
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It iafar from imconmony however, to have the director or otlv'^r project' 
personnel begin to question this budget item as time goes on. After all, 
^*the project appears to be going well and evaluation is not really necessary 
anyway; why don't we just get part of this money transferred to another 
category so we can buy suppli-es with it or pay a classroom aifle? Besides 
that, our evaluator only comes around two or three times a month and his 
dally rate of pay seems to be terribly high, based on the budget allocations.'* 
The fact often overlooked here, of course, is the tremendous amount of 
behind the .scenes work performed by the evaluator that the director never 
sees. This includes such things as development of data collecting instru- * 
ments, designing the evaluation, collecting and processing data, and 
preparation of mid-year and final evaluation reports. 

The **we want to usurp the evaluation dollar** syndrome will ultimately 
be encountered by an evaluator employed by a school district — if he lor 
she stays in the business long enough. The problem is that the evaluator 
is usually not in a decision making role unless he has assumed it by fiat, « 
and thus cannot arbitrate these disputes, they usually must be resolved at 
a higher level in the chain of authority to the schools. The obvious way 
to minimize such events occuring is for the evaluator to do his job so 
effectively there is ho real question as to the services rendered and their 
relationship to the improvement of the program. Evaluation and instruction 
are int^.rtwined. One of our jobs in this emerging profession of vocational 
progr4.li evaluation is to set high standards for ourselves and render 
services which will display the interplay df instruction and evaluation. 

^ The Disinterested Project Director 

The project director usually has other things to do, especially if 
he is running a school, and he may be totally disinterested in the evaluation 
of the particular project assigned to him. This can' work in several ways for 
the evaluator, although in any case his\job becomes much more difficult. 
Since all pertinent administrative directions, state reporting forms, time 
lines, and other key information is usually sent to the director, i: is 
incumbent on him to keep the evaluator informed. If he fails to do this, 
then th^ evaluator must be alert to these things in one way or another and 
must usualj^y end up extrapolating his real role into one of quasi-administra- 
tion and "'picking up the pieces" someone else should be putting together. 
Sometime^ it may be necessary for the evaluator to even take it upon him- 
self to rewrite proposals for continuation of funding, simply because the 
director is disinterested or incompetent to do it himself. This goes 
beyond the evaluator* s ^nonnal" role, but again, each person *s personality 
comes into the picture here. too. The conscientious evaluator who is 
Interested in or believes in the program will not mind going the '^extra 
mile** to see it succeed. In other cases the evaluator may see it as 
detracting from his professional role and will let the project sink. The . 
writer would opt for "going the extra mile" when necessary, but this posture 
has often led him to disputes with colleagues of a different mitid. 

The Interested Project Director 

The first three examples of relationship-situations depicted some- 
what negative viewpoints. We should hasten to point out that there are 
very often extremely positive relationships too. Some project directors 
are Interested in their special programs and want them to succeed — 



either to help youngsters, or to help further tMclr *ownr'car#4f 
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tyttem— or possibly both. In either case, this can be the Ideal setting 
for the evaluator to have an Impact and also to provide a setting. where 
he can bring his own Imagination Into play for a truly creative evaluation. 
If A project Is funded through state or federal sources, It more than 
likely will have a rigidly structured set of forms to be filled out at 
the end of the year. But- the evaluator can add processes/product activi- 
ties over and above the minimum requirements and can become a significant 
member of the project team. This usually works best if the project 
dire>ctor recognizes that the evaluator knows more about the conceptualiza- 
tion, ^he goals, and the, measurement techniques than he, the director does. 
If thn occurs, then the evaluator is normally given a free hand, but the 
interested director will want immediate feed-back at critical points during 
the year so he can make corrective measures where necessary. 

This is where the true worth of an evaluator can be felt ~ helping 
improve the pjrogram Impact on human lives, while it is an actual ongoing 
operation. This long range, but intangible result, will be felt long after 
the year-end evaluation reports have begun to collect dust in a seldom-used 
file cabinet. 

The role of the evaluator seems to be a function of specific project 
reporting requirements, coupled with technical skills, personality factors, 
energy, resourtefulness, willingness to b^ helpful, faith in the project 
being evaluated, constraints of the "system," time limitations, and a host 
of other factors impinging on the total process. Each evaluator will 
consciously or subconsciously find his own mode of operation compatible 
to himself within the framework of the organization that employs him. 

Concluding Remarks • 

This handbook was developed from practical experience in the field 
and was designed to be of day-to-day assistance to the relatively few 
practitioners of vocational program evaluation. The first few»page8 set 
the stage for the "why" of vocational programs and their ultimate need 
for evaluation. The bulk of the manual presents a series of actual case 
studies which provide some basic tools for the on-line evaluator. To 
those who have read ,thc complete handbook it is probably apparent that a 
legal requirement for vocational education evaluation does exist and that 
this legal requirement is sctnehow to be met b^ the various central state 
agencies. These procedures will provide data that may be aggregated 
statewide but, in the authors* opinion, may do little to improve individual 
progra«ns; especially as related to student performance, both in attitude, 
criterion-level achievements, and long-range success, especially compared 
to non-vocational students. These studies (or evaluation) must be done 
at the local level where administrators Ideally are most aware of what 
needs doing and how to go about Hopefully this manual will be 'useful 

in promoting these efforts. 

: STUDY QUESTIONS PART III 

a) Describe in your own words what you feel the role of 
the local evaluator should be - and specify 
impediments you foresee in its attainment. 
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b) Locate a school district that provides its own funds 
(as opposed to federal/state funds) for evaluation 
and analyze what evaluation functions they perform. 



c) Develop an instrument for analyzing the efficacy 
of other test Insti^uments. Try it out on six 
educators and analyze results. 

d) Describe any vocational program you wish and outline 
an evaluation design to measure its effectiveness. 
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